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Foreword 


I was privileged to write the Foreword for the 1993 first edition of Cockpit Resource 
Management. I feel doubly privileged to do the same for this second edition, now re-titled 
Crew Resource Management, a change that reflects many developments that have taken place 
in the intervening time. All of us involved in those early days of “CRM” can rightfully feel 
a sense of pride and satisfaction in what has evolved from an early and comparatively 
rudimentary set of concepts and practices to nearly universally applied precepts that have 
significantly improved the way we conduct training and operations in airplanes, ships, 
medical settings, wildfire management and myriad other previously unimagined appli- 
cations that involve complex human behavior in organizational and team settings. 

In 1993, the verb “to google” didn’t exist. When I wrote this Foreword (mid-February 
2009), “googling” the term “crew resource management” returned 84,300 results, a 
number surely to be much larger by the time this book is published. Interestingly, sub- 
stituting “cockpit” for “crew” in the search term lowered the number of hits by 75% which 
illustrates how significantly the focus has changed from the cockpit to “crews” in diverse 
environments that bear little physical resemblance to cockpits, but share a common reliance 
on complex human performance in a team context for safe and effective functioning. 

In 1993, the accident rate for global scheduled air transport operations was 1.9 hull 
loss accidents per million flights; today, that rate is less than 1.0, a major improvement 
that clearly demonstrates the collective influence of several factors that affect risk and the 
management of risk in our aviation system. Among these are continued improvements 
in the design, manufacture and maintenance of transport category aircraft and power 
plants. Significant improvements in air traffic management, navigation and guidance, 
and weather detection, analysis and information dissemination also have contributed to 
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Foreword 


the improved safety picture. Clearly, too, improvements in human performance brought 
about by the increased understanding and application of the principles of CRM have 
played a major role in reducing accidents in aviation. Much of this progressive devel- 
opment is attributable to the editors and authors of these two editions, and the work of 
others whose contributions are described in this volume. 

In 1993, I used two major air transport accidents to illustrate the reduction of risk in 
airline operations made possible by the early introduction of CRM concepts—the 1972 
Lockheed L-1011 accident in the Florida Everglades, and the 1989 McDonnell Douglas 
DC-10 accident at Sioux City, Iowa. The first accident claimed the lives of all 163 
passengers and 13 crewmembers after the flight crew became distracted while changing 
a burned-out indicator light and allowed the aircraft to descend into the swamp. In the 
second accident, nearly two thirds of the total of 296 passengers and crew survived in 
large part because the crew successfully applied the principles of CRM to manage what 
otherwise would have been a non-survivable event due to total loss of flight controls. 

As I write this Foreword, only a few weeks have passed since what some have termed 
“the Miracle on the Hudson.’ All passengers and crew survived the ditching of an Airbus 
A320 in the Hudson River due to a double engine failure consequent to multiple bird 
strikes shortly after takeoff. Although the NTSB report on this accident is many months 
away, it appears that what could have been a major disaster for those aboard, and 
potentially many on the ground, was instead a tale of what went right. In no small part 
this outcome was due to the exquisite management of all available resources by the 
cockpit and cabin crewmembers and by the ground forces that responded to the 
ditching. Again, the fortunate outcome of this event represents the confluence of many 
factors, but it is very clear that none of those would have made much of a difference had 
the flight crew not executed a successful ditching, and, subsequently and in close concert 
with the cabin crew, evacuated all 155 persons on the aircraft. This accident seems to 
represent the highest form of human performance—CRM at its very best. 

In 1993, I concluded (CRM) is an exciting story, and one which offers great 
personal gratification. There are few more rewarding efforts than those which result in 
the saving of lives.” In the intervening years, the exciting story and its then only 
imagined benefits have generated nearly universal application of CRM principles in 
virtually thousands of settings. This is a direct result of evolutionary developments in 
concept and practice honed by a multitude of dedicated researchers and practitioners. 

Still, it remains a story of enormous personal gratification and reward—countless 
lives have undoubtedly been saved by the collective efforts of those whose works are 
chronicled here. 

John K. Lauber 
Vaughn, WA 


Preface 


In 1993, Cockpit Resource Management (CRM) was celebrated as the convergence of 
a concept, an attitude and a practical approach to pilot training. Equally important was 
the convergence and enthusiastic support of the research community, aviation regula- 
tors, transport operators and pilot organizations. CRM was maturing, implementing 
and continuing to develop all at the same time. 

It was always said that if CRM succeeded, it would disappear as stand-alone training 
as it became fully integrated into an airline’s training program. As early as 1990 the 
Federal Aviation Administration (FAA) provided a mechanism for achieving just that, in 
the form of the Advanced Qualification Program (AQP). But CRM grew in many other 
directions as well. Fifteen years later, CRM concepts have endured not only by dis- 
appearing into the fabric of training, but by expanding the team concept, evolving into 
new applications and now integrating itself into an even higher level of safety and quality 
assurance goals. 

Even in 1993, it was evident that CRM was being applied beyond the cockpit and 
we acknowledge that CRM more appropriately stands for Crew Resource Manage- 
ment. While we will continue to focus on CRM in the cockpit in this edition, we want 
to emphasize that the concepts and applications provide generic guidance and lessons 
learned for a wide variety of “crews” in the aviation system and in the complex, high- 
risk operations of many non-aviation settings. 

In the late 1970s, when our late colleague H. Patrick Ruffell Smith launched his 
classic study of flight crew performance in a Boeing 747 simulator, he could not have 
dreamed of what would be inspired by that project. The experiment originally inves- 
tigated pilot vigilance, workload and response to stress. What is a great testament to that 
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early research is that we continue to make effective use of the simulator to investigate 
vigilance (situational awareness), workload management and response to stress—and 

a host of human factors affected by the continually evolving airspace system. Ruffell 
Smith opened the door to researching human factors and to a useful methodology for 
assessing crew performance as well as the reliability of instructor and evaluator 
judgments. 

CRM training, like any new approach to a well-established, tradition-bound 
enterprise, was not universally acclaimed in its early years. Many airline managers 
dragged their feet; they claimed that they were doing it anyway, just not under the name 
of CRM. And besides, who had any proof that the new training was effective? The FAA 
viewed the field with a degree of skepticism in the beginning, in spite of a string of 
recommendations from the National Transportation Safety Board (NTSB) that CRM 
training be required of the nation’s airlines. 

In our original 1993 edition of Cockpit Resource Management, it was clear that 
momentum was taking hold, not only in US commercial aviation, but in the military 
and abroad. While AQP was still under development, US and international operators, 
pilot organizations, investigators, regulators, researchers and others in the industry 
became an active CRM community that experimented with training approaches and 
shared its results. In the spirit of its successful collaboration, this edition incorporates 
a mixture of US and non-US researchers, operators and regulators; our authors 
personally remember the beginnings of CRM and helped to support and develop new 
directions and refinements to build what CRM is today. 

Crew Resource Management, 2nd Edition, consists of three main sections: (1) Nature of 
CRM, (2) CRM Training Applications and (3) CRM Perspectives. A short final section, 
Conclusions, provides summary observations about the current state-of-the-practice 
and thoughts of the future. Following are brief descriptions of three main sections. 

Part 1: Nature of CRM contains an introduction and discussion of familiar CRM 
concepts and skills, such as the teamwork, communication and decision-making. 
Much of the original research and early initiatives are preserved in these pages and it 
is exciting to see how far the concepts have grown, not just from a theoretical 
perspective but how they have matured into useful training methods and expanded in 
directions we had not imagined. 

Part 2: CRM Training Applications contains chapters that describe some of the 
many innovations that have grown from CRM training. It introduces tools that support 
CRM training development and performance assessment as well as audit tools that are 
used in line operations. While there are numerous examples of CRM adapted to other 
airline teams (e.g. maintenance, flight attendants, dispatch), and to teams outside 


Preface 


aviation (e.g. spaceflight, medical, rail), there are too many to cover them all. However, 
we devote a couple of chapters to review some of these developments. 

Part 3: CRM Perspectives consists of chapters that illustrate the impact of CRM 
when implemented. While flight training departments in both commercial and military 
transport operations were the original practitioners of CRM training, concepts and skills 
were adopted in many countries and also influenced the way in which regulators and 
investigators understood and analyzed human performance. This section discusses some 
of these perspectives and describes a few of the many ways CRM is implemented in 
different airline/transport cultures. Although the basic CRM topics in Part 1 have 
a generic quality, Part 3 shows how common topics are tailored to fit different 
organizations. 

Corresponding to the breadth of CRM topics, applications and perspectives, we 
write to a global audience including aviation training practitioners, managers, corporate 
decision-makers, regulators, investigators and researchers. In addition, we hope that 
other airline departments, as well as non-aviation industries, will find topics of interest 
and usefulness in developing CRM tools and programs for their own work settings. We 
believe CRM presents a success story as it has transcended its own training roots and 
followed a path of adaptation and expansion that addresses larger, safety management 
objectives. It is a model which—in spite of cultural barriers, economic setbacks and 
bureaucratic complexities—has become a household word in aviation. 

We thank our authors for a job well done and for helping to preserve this docu- 
mentation of CRM history and lessons learned. In addition, we are grateful to the staff at 
Elsevier for their very professional help and encouragement. Finally, we dedicate this 
volume to the thousands of flight crews throughout the world whose participation and 
experiences continue to provide the reason that CRM succeeds. 

Barbara G. Kanki 
Robert L. Helmreich 
José Anca 
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Introduction 


Section 1.1 of this chapter is the introductory chapter of the 1993 edition of the 
book. This reprint is important for the reader because it covers the antecedents and 
history of CRM from 1978 until 1992. Some of the predictions for the future of CRM 
have been borne out while others have not. Fifteen years ago, CRM was not universally 
accepted by the pilot community: it was sometimes decried as charm school, 
psychobabble, and attempted brainwashing by management and some of these criticisms 
had merit. The evolution of CRM is covered through its third generation. 

Section 1.2, CRM Redux, covers the fourth, fifth and the current sixth generation 
which focuses on the threats and errors that must be managed by crews to ensure 
safety in flight. 


1.1. The Evolution and Growth of CRM 


1.1.1. Introduction 


One of the most striking developments in aviation safety during the past decade has been 
the overwhelming endorsement and widespread implementation of training programs 
aimed at increasing the effectiveness of crew coordination and flightdeck management. 
Civilian and military organizations have developed programs that address team and 
managerial aspects of flight operations as complements to traditional training that stresses 
the technical, “‘stick-and-rudder” aspects of flight. The original, generic label for such 
training was cockpit resource management, but with recognition of the applicability of the 
approach to other members of the aviation community including cabin crews, flight 
dispatchers, and maintenance personnel, the term crew resource management (CRM) is 
coming into general use. 

Just as CRM has evolved from “cockpit” to “crew” over its short history, the field of 
human factors has similarly changed in its scope. From an initial marriage of engineering 
and psychology with a focus on “knobs and dials,” contemporary human factors has 
become a multidisciplinary field that draws on the methods and principles of the 
behavioral and social sciences, engineering, and physiology to optimize human 
performance and reduce human error (National Research Council, 1989). From this 
broader perspective, human factors can be viewed as the applied science of people working 
together with devices. Just as the performance and safety of a system can be degraded 
because of poor hardware or software design and/or inadequate operator training, so too 
can system effectiveness be reduced by errors in the design and management of crew-level 
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tasks and of organizations. CRM is thus the application of human factors in the aviation 
system. John K. Lauber (1984), a psychologist member of the National Transportation 
Safety Board (NTSB), has defined CRM as “using all available resources—information, 
equipment, and people—to achieve safe and efficient flight operations” (p. 20). CRM 
includes optimizing not only the person—machine interface and the acquisition of timely, 
appropriate information, but also interpersonal activities including leadership, effective 
team formation and maintenance, problem-solving, decision-making, and maintaining 
situation awareness. Thus training in CRM involves communicating basic knowledge 
of human factors concepts that relate to aviation and providing the tools necessary to 
apply these concepts operationally. It represents a new focus on crew-level (as opposed 
to individual-level) aspects of training and operations. 

This chapter’s title inquires why an industry would embrace change to an approach 
that has resulted in the safest means of transportation available and has produced 
generations of highly competent, well-qualified pilots. In seeking the answer, we 
examine both the historic, single-pilot tradition in aviation and what we know about the 
causes of error and accidents in the system. These considerations lead us to the 
conceptual framework, rooted in social psychology, that encompasses group behavior 
and team performance. In this context we can look at efforts to improve crew 
coordination and performance through training. Finally, we discuss what research has 
told us about the effectiveness of these efforts and what questions remain unanswered. 


1.2. The Single-Pilot Tradition in Aviation 


The evolution of concern with crew factors must be considered in the historical context 
of flight. In the early years, the image of a pilot was of a single, stalwart individual, white 
scarf trailing, braving the elements in an open cockpit. This stereotype embraces 
a number of personality traits such as independence, machismo, bravery, and calmness 
under stress that are more associated with individual activity than with team effort. It is 
likely that, as with many stereotypes, this one may have a factual basis, as individuals with 
these attributes may have been disproportionately attracted to careers in aviation, and 
organizations may have been predisposed to select candidates reflecting this prototype. 
As aircraft grew more complex and the limitations and fallibility of pilots more 
evident, provision was made for a co-pilot to provide support for the pilot, to reduce 
individual workload and decrease the probability of human error. However, these 
additional crewmembers were initially perceived more as redundant systems to be used 
as backups than as participants in a team endeavor. Ernest K. Gann (1961) and other 
pioneers of air transport have documented the distinctly secondary role played by the 
co-pilot in early airline operations. 
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The tradition in training and evaluation has similarly focused on the individual pilot 
and his or her technical proficiency (Hackman & Helmreich, 1987). This begins with 
initial selection and training, which have historically used aptitude and performance 
standards developed for single-pilot operations. Indeed, the first critical event in a pilot’s 
career is the solo flight. Even in multipilot operations, the major emphasis continues to 
be on evaluting the individual proficiency of crewmembers. Regulations surrounding 
the qualification and certification of pilots reinforce these practices and can even result in 
negative training. For example, in crewmembers are cautioned not to provide assistance 
to pilots whose proficiency is being evaluated, a model of individual instead of team 
action is being reinforced. Indeed, in 1952 the guidelines for proficiency checks at one 
major airline categorically stated that the first officer should not correct errors made by 
the captain (H. Orlady, personal communication cited in Foushee & Helmreich, 1988). 
The critical point is that the aviation community has operated on the assumption that 
crews composed of able and well-trained individuals can and will operate complicated 
aircraft in a complex environment both safely and efficiently. 


1.3. Human Error in Flight Operations 


The introduction of reliable turbojet transports in the 1950s was associated with 

a dramatic reduction in air transport accidents. As problems with airframes and engines 
diminished, attention turned to identifying and eliminating other sources of failure in 
flight safety. Figure 1.1 gives statistics on the causes of accidents from 1959 through 
1989, indicating that flightcrew actions were casual in more than 70% of worldwide 


Figure 1.1 Primary causes of hull loss accidents (excluding military and sabotage): 
worldwide commercial jet fleet, 1959-1989. Data from Boeing Aircraft Company 
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accidents involving aircraft damage beyond economical repair. Recognition of this 
human performance problem stimulated a number of independent efforts to understand 
what the term “pilot error” encompassed and what could be done to reduce it. 

The formal record of investigations into aircraft accidents, such as those conducted 
by the NTSB, provides chilling documentation of instances where crew coordination 
has failed at critical moments. 


m A crew, distracted by the failure of a landing gear indicator light, failing to 
notice that the automatic pilot was disengaged and allowing the aircraft to 
descent into a swamp. 


m A co-pilot, concerned that take-off thrust was not properly set during a departure 
in a snowstorm, failing to get the attention of the captain with the aircraft 
stalling and crashing into the Potomac River. 


m A crew failing to review instrument landing charts and their navigational position 
with respect to the airport and further disregarding repeated Ground Proximity 
Warning System alerts before crashing into a mountain below the minimum 
descent altitude. 


m A crew distracted by nonoperational communication failing to complete 
checklists and crashing on take-off because the flaps were not extended. 


m A breakdown in communication between a captain, co-pilot, and Air Traffic 
Control regarding fuel state and a crash following complete fuel exhaustion. 


m A crew crashing on take-off because of icing on the wings after having inquired 
about de-icing facilities. In the same accident the failure of a flight attendant 
to communicate credible concerns about the need for de-icing expressed by pilot 
passengers. 


The theme in each of these cases is human error resulting from failures in 
interpersonal communications. By the time these accidents occurred, the formal study 
of human error in aviation had a long tradition (e.g., Fitts & Jones, 1947; Davis, 1948). 
However, research efforts tended to focus on traditional human factors issues 
surrounding the interface of the individual operator with equipment. This type of 
investigation did not seem to address many of the factors identified as causal in jet 
transport accidents, and researchers began to broaden the scope of their inquiry. 

In the United States, a team of investigators at NASA—Ames Research Center began 
to explore broader human factors issues in flight operations. Charles Billings, John 
Lauber, and George Cooper developed a structured interview protocol and used it to 


Chapter 1 » Why CRM? Empirical and Theoretical Bases of Human Factors Training 


gather firsthand information from airline pilots regarding human factors in crew 
operations and “pilot error” accidents. At the same time, George Cooper and Maurice 
White analyzed the causes of jet transport accidents occurring between 1968 and 1976 
(Cooper, White, & Lauber, 1980), while Miles Murphy performed a similar analysis of 
incidents reported to NASA’s confidential Aviation Safety Reporting System (Murphy, 
1980). The conclusion drawn from these investigations was that “pilot error” in 
documented accidents and incidents was more likely to reflect failures in team 
communication and coordination than deficiencies in “stick-and-rudder” proficiency. 
A number of specific problem areas were identified, including workload management 
and task delegation, situation awareness, leadership, use of available resources including 
other crewmembers, manuals, air traffic control, interpersonal communications 
(including unwillingness of junior crewmembers to speak up in critical situations), and 
the process of building and maintaining an effective team relationship on the flightdeck. 

In Europe, Elwyn Edwards (1972) drew on the record of accident investigation and 
developed his SHEL model of human factors in system design and operations. The 
acronym represents software, usually documents governing operations; hardware, the 
physical resources available; liveware, consisting of the human operators composing the 
crew; and environment, the external context in which the system operates. Elaborating his 
model to examine the functioning of the liveware, Edwards (1975) defined a new 
concept, the trans-cockpit authority gradient (TAG). The TAG refers to the fact that 
captains must establish an optimal working relationship with other crewmembers, with 
the captain’s role and authority neither over- nor underemphasized. 

In the operational community in the early 1970s, Pan American World Airways 
management became concerned about crew training issues following several “pilot 
error” accidents in the Pacific. In 1974, a flight operations review team headed by David 
D. Thomas, retired Deputy Administrator of the Federal Aviation Administration 
(FAA), examined all aspects of flightcrew training and made a number of significant 
recommendations. The foremost of these was to utilize “crew concept training.” Under 
this approach, both simulator training and checking were to be conducted not as single- 
pilot evolutions but in the context of a full crew conducting coordinated activities. At 
the same time, Pan Am manuals were revised to incorporate crew concepts and to 
explain more completely responsibilities for team activities and communications. These 
actions represented a fundamental change in the operating environment and provided an 
organizational framework for more effective crew coordination. Although the focus in 
training was now on crew activities, the shift was not accompanied by a program of 
formal instruction in communications and coordination. Crewmembers were mandated 
to operate as effective teams but were left to develop means of achieving this goal 
without formal guidance and instruction. 
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Identifying crew-level issues as central to a high proportion of accidents and 
incidents was a significant achievement in the process of understanding the 
determinants of safety in flight operations. However, development of successful 
strategies to improve crew performance requires an understanding of the determinants 
of group behavior and how they can be influenced. In the following section we 
describe a model of group processes and performance and its implications for training 
and organizational actions. 


1.4. Group Processes and Performance in the 


Aviation Environment 


The study of group behavior has historically been the province of social psychology and 
provides the conceptual basis for the three-factor model of the determinants of group 
performance we presented in an earlier discussion of flightcrew interaction and 
performance (Foushee & Helmreich, 1988; McGrath, 1964). Subsequent research has 
enabled us to expand and refine the model, and we present it as a framework for 
discussing issues surrounding CRM training. The model defines three major 
components of group behavior: input factors, which include characteristics of individuals, 
groups, organizations, and the operational environment; group process factors, which 
include the nature and quality of interactions among group members; and outcome factors, 
which include primary outcomes such as safety and efficiency of operations and 
secondary outcomes such as member satisfaction, motivation, attitudes, and so on. The 
underlying assumption of the model is that input factors both provide the framework 
and determine the nature of group processes that lead, in turn, to the various outcomes. 
Figure 1.2 shows the three factors and their interrelationships. A central feature of the 
model is feedback loops among the factors. Outcomes (right side of figure; either 
positive or negative) may change components of input factors (left side; e.g., attitudes 
and norms), and these changes may alter subsequent group processes (middle) and 
outcomes. Outcomes may theoretically also influence group processes without being 
directly mediated by input factors. It is the iterative nature of the factors determining 
group performance that makes its study both complex and challenging. 


1.4.1. Outcome Factors 


Primary outcome factors are readily recognizable and relatively easily quantifiable. In 
flight operations safety is paramount, but the efficient completion of missions and 
compliance with organizational and regulatory requirements are also important. 
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Figure 1.2 Flightcrew performance model 
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Both experience and training can create changes in crew attitudes and norms 
regarding appropriate flightdeck management. The quality of group processes, 
influenced by organizational, group, regulatory, and environmental factors, 
determines the satisfaction crews experience with operations and their motivation 
for future operations. 

Outcome factors form the criteria against which the impact of interventions such as 
training or organizational policy changes are measured. While the most compelling 
measure of effectiveness in aviation would be a decrease in the frequency of accidents, 
such events are (happily) already so infrequent that reliable statistical evidence can only 
be found by aggregating data over extremely long periods of time. Accordingly, criteria 
of group performance need to be drawn from surrogate measures such as records of 
operational errors, expert ratings of crew effectiveness, and measures of attitude and job 
satisfaction. 


1.4.2. Input Factors 


A number of qualitatively different variables form the inputs to group processes. 
These have multiple components that, singly and in combination, influence the way 
teams interact. Figure 1.3 expands the input factors portion of the model to include 
lower-order variables that have a demonstrated influence on group processes and 
outcomes. 
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Figure 1.3 Flightcrew performance model: expanded input factors. 
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Individual factors 


Consideration of a flightcrew’s job in today’s airspace brings to mind a number of 
background or input factors that can influence the effectiveness of crew activities even 
before an engine is started. Teams are composed of individuals who bring to the flightdeck 
their knowledge, skills, personalities, motivation, and physical and emotional states. Each 
of these characteristics has been identified as causal in one or more aircraft accidents. 
Physical condition includes fatigue, which can undermine vigilance in a 
knowledgeable and motivated pilot. Emotional state is determined by a variety of life 
stresses (for example, marital discord or worries about the financial condition and 
viability of an airline) that cannot be left at the gate and can subtly undermine 
effectiveness. Aptitude (including intelligence and psychomotor skills) has long been 
recognized as critical to success as a pilot, and selection has emphasized these attributes. 
Recent research has also confirmed that personality factors are significant determinants 
of individual and team performance. A full-mission simulation study was run with 
volunteer, three-person crews in the NASA—Ames Boeing 727 simulator. The study 
explored the impact of leader personality factors on crew performance (Chidester, 
Kanki, Foushee, Dickinson, & Bowles, 1990). Crewmembers participating in the study 


12 


Chapter 1 » Why CRM? Empirical and Theoretical Bases of Human Factors Training 


were pretested on a personality battery that had been validated as predictive of flightcrew 
behavior (Chidester, Helmreich, Gregorich, & Geis, 1991). Three experimental groups 
were composed on the basis of the captain’s personality constellation. One group was led 
by captains high on both goal orientation and interpersonal skills. A second group had 
captains who were high on goal orientation but relatively low on the interpersonal 
dimension. The third group was led by captains who were quite low on both goal 
orientation and positive interpersonal dimensions. 

Each crew flew five complete flight segments spread across two days. On two of the 
legs, mechanical malfunctions occurred which were compounded by poor weather 
conditions at the destination airport. Crew performance was rated by expert observers, 
and technical errors were coded from computer records and videotapes of the flights. 
The data showed significant differences in performance between groups that could be 
attributed to the leader’s personality. Crews led by captains high in both achievement 
needs and interpersonal skills performed uniformly well across all segments. In contrast, 
crews led by captains low on both of these dimensions were significantly less effective 
across all flights. Those in the third group, with captains high in achievement needs but 
low in interpersonal traits, were given poorer performance ratings initially but improved 
substantially by the fifth leg. One interpretation of this finding is that crews in this 
condition learned over time how to adapt to this difficult but motivated type of leader. 
The point relevant to this discussion is that a single input factor (personality) can be 
isolated as an influence on the performance of a well-trained and qualified crew in 
a controlled research setting. 

Attitudes serve as guides for behavior and are another of the input factors that crews 
bring to the flightdeck. The Cockpit management attitudes questionnaire (CMAQ, Helmreich, 
1984; Helmreich, Wilhelm, & Gregorich, 1988) is a 25-item, Likert-scaled battery that 
allows quantification of attitudes regarding crew coordination, flightdeck management, 
and personal capabilities under conditions of fatigue and stress. Attitudes measured by 
the CMAQ have been validated as predictors of outcome factors in the form of expert 
ratings of performance in line operations (Helmreich, Foushee, Benson, & Russini, 
1986), thus demonstrating the linkage between input and outcome factors. Measures such 
as the CMAQ can be used both to assess input factors in organizations and as measures 
of outcomes to determine whether programs such as CRM can change attitudes. 


Group factors 


Crews are composed of individuals who bring with them all the attributes noted above. 
They may be cohesive and effective or divisive, rancorous, and ineffectual depending 
on the mix of individuals and their states that comes together at any given time. The 
climate that develops in a group is multiply determined by the characteristics of 
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individual members, by the structure imposed by the formal and informal norms of the 
organization, and by the quality and style of leadership present. Because of the many 
individual and group factors identified, research into these issues and their effects is 
difficult and time-consuming. As a result there is not an extensive literature on the 
outcome effects of systematically varying multiple individual- and group-level variables, 
especially in the aviation environment. 


Organizational factors 


The culture of an organization is a critical input factor. If an organization sanctions 
individual actions rather than team coordination, both processes and outcomes are likely 
to have a very different flavor from those in organizations that stress crew actions and 
responsibility. The level of training and type of formal evaluation given to crews are also 
influential. Manuals and formal procedures also form part of the operational setting, as 
do the resources that the organization has and makes available for crews (including crew 
scheduling practices, maintenance support, flight planning, dispatching, etc.). 

Another NASA simulation study examined the performance implications of several 
individual- and group-level factors. Foushee, Lauber, Baetge, & Acomb (1986) examined 
the interactions and performance of experienced two-person jet transport crews flying 
a realistic scenario in a Boeing 737 simulator. NASA was directed by the U.S. Congress to 
investigate the operational significance of pilot fatigue—an individual factor driven by 
organizational and regulatory practices. The experimental design reflected this concern 
and divided crews into two groups, pre-duty (defined as flying the scenario after 
a minimum of two days off as if it were the first leg ofa three-day trip) and post-duty (flying 
the scenario as the last segment of a three-day trip). The scenario was characterized by 
poor weather conditions that necessitated an unexpected missed approach that was 
complicated by a hydraulic system failure. Following the hydraulic failure, crews were 
faced with a high-workload situation involving the selection of an alternate destination 
while coping with problems such as the requirement to extend gear and flaps manually and 
fly an approach at higher than normal speed. 

Crews in the post-duty condition had less pre-simulation sleep and reported 
significantly more fatigue, as expected from the research design. The surprising finding, 
however, was that fatigued crews were rated as performing significantly better and made 
fewer serious operational errors than the rested, pre-duty crews. This finding was 
counterintuitive but had major implications relevant to the importance of team 
formation and experience. By the nature of the scheduling of flight operations, most 
crews in the post-duty condition had just completed three days of operations as a team, 
while those in the pre-duty condition normally did not have the benefit of recent 
experience with the other crewmember. When the data were reanalyzed on the basis of 
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whether or not crews had flown together recently, the performance differences became 
even stronger. The findings suggest that crew scheduling practices that result in 
continuing recomposition of groups and a need for frequent formation of new teams can 
have significant operational implications. For example, three recent takeoff accidents in 
the United States (one involving a stall under icing conditions, one an aborted takeoff 
with an over-run into water, and one a runway collision after the crew became lost in 
dense fog) involved crews paired together for the first time.' The implications of crew 
pairings are discussed further in the chapter by Hackman. 


Environmental factors 


Weather conditions constitute an environmental input factor outside the control of 
flightcrews. The ability of organizations and the government to provide accurate, timely 
information on weather constitutes one of the factors governing both group processes 
and outcomes. The physical condition of the aircraft (including inoperative equipment, 
etc.) also determines part of the field in which the crew must operate as does the 
availability and quality of navigational aids. 


Regulatory factors 


Regulatory practices also influence the nature of crew interaction and performance. For 
example, the “sterile cockpit” rule in the U.S. proscribes non-operational 
communications below 10,000 feet. As described above, the focus of regulation has been 
on individual training and evaluation, and this has been echoed in organizational policies 
(recall the prohibition on first officers correcting captain’s mistakes during proficiency 
checks). Ambiguity in regulations can also impact crews’ decisions and actions. If the 
regulations governing an operation are unclear, responsibility shifts to the organization 
that can direct operations to meet operational goals and to the captain who must take 
ultimate responsibility for decisions regarding the safety of flight. 


1.4.3. A Case Study: The Interplay of Multiple 
Input Factors in a Crash 


Investigation of the human factors surrounding the crash of a Fokker F-28 on takeoff in 
Canada demonstrates the interplay of input factors at the regulatory, organizational, 





' One involved a DC-9 taking off in a snowstorm at Denver, the second a rejected take 
off by a B-737 at New York-LaGuardia, and the third a DC-9 that erroneously taxied 
onto the active runway and collided with a B-727 taking off. 
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environmental, and individual levels. In this accident it can be seen how all of these can 
intersect to create an operational environment that fails to provide needed safeguards 
against pilot error (Helmreich, 1992; Moshansky, 1992). On a snowy winter afternoon 
the crew of Air Ontario Flight 1363 attempted a takeoff from Dryden, Ontario, with an 
accumulation of snow and ice on the wings and crashed because the aircraft could not 
gain enough lift to clear trees beyond the end of the runway. In the crash and resulting 
fire, 29 passengers and crewmembers, including both pilots, were killed. In attempting 
to understand how a crew with many years of experience operating in the severe winter, 
weather of northern Ontario could make such a serious operational error, a number of 
input factors were uncovered which, operating in concert, set the stage for a tragically 
wrong decision. 

At the environmental level, the weather was poor and deteriorating, forcing the crew 
to select distant alternate landing sites and to carry extra fuel. Because of the poor 
weather, the flight was operating more than an hour late and was full, operating at 
maximum gross weight. The aircraft itself had a number of mechanical problems, the 
most serious of which was an inoperative auxiliary power unit (APU). With an 
inoperative APU, it was necessary to keep an engine running during stops at airports 
without ground start capabilities. Dryden had no such facilities. 

At the regulatory level, the Canadian regulations regarding de-icing prohibited an 
aircraft from commencing a flight “when the amount of frost, snow, or ice adhering to 
the wings, control surfaces, or propeller of the aeroplane may adversely affect the safety 
of flight” (Moshansky, 1989).? The problem facing the crew under existing regulations 
was how, under time and operational pressures, to determine what constituted enough 
contamination to “adversely affect” safety of flight. The regulation as written made the 
takeoff decision at the captain’s discretion and, at the same time, failed to provide 
safeguards against personal and organizational pressures to complete the mission at all 
costs. 

The regulatory agency’s surveillance of the airline had not focused on the newly 
initiated jet operation. While an audit of the airline’s operations had been completed 
during the preceding year, the audit did not include the F-28 operation. A more 
complete examination might have revealed procedural and organizational discrepancies 
in the F-28 operation, as noted below. 

A number of organizational factors served to increase the stress level of the crew. The 
airline had just begun operating jet transports and had little operational experience with 





* In response to a recommendation by the Commission of Inquiry into the crash, the 
regulation was changed to prohibit operation with any contamination of lifting 
surfaces. 
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this type of equipment. Initial crews for the Fokker had been trained at two different 
US. airlines before the operation was initiated. The airline had not developed its own 
operating manuals, and some crewmembers were carrying manuals from one airline and 
others from another. The organization had not developed an approved minimum 
equipment list (MEL) specifying what equipment could be inoperative in normal 
passenger operations. Dispatchers had received only minimal training for this type of 
aircraft and were experienced only with small propeller-driven equipment. The flight 
release for the day of the accident contained a number of errors. In sum, the crew was 
operating without a high level of organizational support and resources. 

The airline itself was the product of the merger of two regional airlines with very 
different operational cultures. One had operated in the north of Canada as what was 
often called a “bush” operation. The other had operated in southern Ontario in a more 
traditional airline environment. The chief pilot of the Fokker fleet had come from the 
northern operation and had himself had two serious incidents involving take-offs with 
ice on the wings—experiences that had earned him the nickname of “Iceman.” These 
practices suggest the possibility that norms and pressures existed to operate with wing 
contamination. The ambiguous regulation (see p. 13) provided no safeguard against such 
norms and pressures. 

As individuals, both crewmembers had extensive experience in Canadian operations. 
The captain had more than 24,000 flight hours and the co-pilot more than 10,000. 
However, neither had much experience in jet transport operations, the captain having 
accumulated 81 hours in the F-28 and the first officer 65. The captain had been a chief 
pilot and instructor and was known for adherence to procedures. The first officer was 
a former captain described as having a somewhat abrasive personality. He also had 
a history of difficulties in completing some stick-and-rudder maneuvers and had 
required additional supervision and training before qualifying in new aircraft. 

As a group, the crew had only flown together for two days. The fact that the crew 
lacked operational familiarity with each other and with the aircraft, along with the fact 
that both were accustomed to flying as captains, may have influenced the processes 
surrounding their conduct of the flight. In addition, the captain came from the more 
structured southern airline, while the first officer’s experience was in the less formal 
northern operation. 

When the aircraft landed to pick up passengers at Dryden, the crew faced a complex 
and stressful situation. Weather was deteriorating further, with heavy snow falling. 
Refueling was needed before departure, but this would necessitate keeping an engine 
running because of the inoperative APU. The cabin manual prohibited refueling with 
passengers aboard and an engine running, but the cockpit manuals were silent on this 
issue. The flight attendants were not alerted to the need to refuel with an engine 
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running. The manufacturer’s manual further prohibited de-icing with an engine 
running because of possible ingestion of fluid into the powerplant. The flight was falling 
further behind its schedule, and many passengers were facing the prospect of missing 
connecting flights if there was an additional delay for de-icing. 

Faced with these contingencies, the crew chose to refuel with passengers aboard and 
an engine running. It is known that the captain considered de-icing, because he inquired 
about the availability of equipment and was told that it could be provided. Ultimately, 
however, the crew chose to take off without de-icing. Having reached this decision, 

a further environmental factor intervened in the form of a small plane, flying under 
visual flight rule (VFR) conditions, which made an emergency landing, causing 
additional delay until the runway was cleared. 

There were also several experienced pilots, including two airline captains, seated as 
passengers in the main cabin. They survived and testified to being aware of the need for 
de-icing and the associated threat to safety. One of them expressed his concerns about 
icing to the lead flight attendant but was told (falsely) that the aircraft had automatic 
de-icing equipment. These credible concerns were never communicated to the 
flightdeck by the flight attendants. This failure in communication is understandable in 
light of organizational norms regarding cabin-cockpit communication on safety issues. 
One of the managers of flight attendant training testified that flight attendants were 
trained not to question flightcrews’ judgment regarding safety issues. 

Because the cockpit voice recorder was destroyed in the fire following the crash, it is 
impossible to reconstruct the interaction processes that led to the decision to depart 
Dryden without de-icing. While there was unquestionably human error in that 
decision, to stop at this conclusion would be to ignore the extent to which the input 
factors set the stage for the outcome. 


1.4.4. Group Process Factors 


Group process factors have historically been the least studied and least understood 
aspects of team performance. Much of the research that has been done, especially in 
operational settings, has looked at input and outcome factors, leaving the intervening 
process as a block box (e.g., Foushee, 1984; Foushee & Helmreich, 1988; Hackman & 
Morris, 1975). Input factors are manifested in the types of interactions that occur when 
individuals and machines come together to execute complex tasks in a complex 
environment. The fact that process variables have been largely ignored in research does 
not indicate a lack of awareness of their importance; rather, it reflects the difficulty of 
conceptualizing and measuring them. There are a number of important and theoretically 
interesting questions regarding flightcrew group processes: (1) How do individuals come 
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together as strangers and forge a cohesive team that can operate effectively after only 
a brief acquaintance? (2) How is team workload managed and delegated? (3) What 
means are used to integrate ambiguous and incomplete data to reach optimum decisions? 
(4) How does stress induced by fatigue, emergencies, and personal experiences influence 
the way teams communicate and operate? (5) What is the nature of effective and 
ineffective leadership among flightcrews? 

Group processes are manifested primarily through verbal communications, and 
these provide the record that we can use to understand how teams function in flight 
operations. Fortunately, there is a growing base of empirical research on group 
processes among flightcrews, much of it from experimental flight simulations. As 
Foushee (1984) has pointed out, modern flight simulators provide investigators with 
an extraordinarily useful research setting. Simulation provides high experimental 
realism including visual, motion, and auditory cues. Major aspects of flight operations 
can be reproduced, including mechanical problems, weather, air-to-ground 
communications, and cabin—cockpit interactions. Flight-plans can be generated and 
normal and abnormal operations between real airports simulated. Having experienced 
crews “fly” familiar equipment using normal procedures and manuals further 
enhances the external validity and generality of findings from simulations. Participants 
in experimental simulations report that realism is high and that motivation is 
comparable to that in regular line operations. Because simulators can be programmed 
to provide an identical operating environment for each crew, it is possible to gain 
statistical power by exposing many crews to the same conditions. To isolate causal 
factors, operational factors can be experimentally varied for different subgroups of 
participants: for example, the manipulation of recent experience in the simulation 
addressing fatigue. The simulator computer provides a record of the crew’s physical 
actions controlling the aircraft, while video and audio recordings capture the 
interpersonal aspect of flight. The simulations described earlier have yielded important 
data on the impact of input factors such as operational experience and personality and 
have also allowed quantification of the processes involved. 

Although not designed as a study of group processes, an experimental simulation 
sponsored by NASA and conducted by the late H. Patrick Ruffell Smith (1979) is 
a powerful demonstration of the operational significance of crew interactions. Eighteen 
airline crews flew a two-segment flight in a Boeing 747 simulator. The scenario 
consisted of a short flight from Washington, D.C., to John F Kennedy Airport in New 
York and a subsequent leg from New York to London. After departing from New York, 
the crew experienced an oil pressure problem that forced them to shut down an engine. 
Because the flight could not be completed with a failed engine, the crew had to decide 
where to land. This decision was complicated by the further failure of a hydraulic 
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system, deteriorating weather at possible landing sites, complex instructions from air 
traffic control, and a cabin crewmember who repeatedly requested information and 
assistance from the flightdeck at times of high workload. The study showed a remarkable 
amount of variability in the effectiveness with which crews handled the situation. Some 
crews managed the problems very well, while others committed a large number of 
operationally serious errors, including one miscalculation of more than 100,000 pounds 
in dumping fuel. The primary conclusion drawn from the study was that most problems 
and errors were induced by breakdowns in crew coordination rather than by deficits in 
technical knowledge and skills. For example, many errors occurred when individuals 
performing a task were interrupted by demands from other crewmembers or were 
overloaded with a variety of tasks requiring immediate action. In other cases, poor 
leadership was evident and resulted in a failure to exchange critical information in 

a timely manner. 

The cockpit voice data from the study were subsequently analyzed by Foushee & 
Manos (1981) to quantify the processes related to variability in group performance. 
Their approach grew out of social psychological research into information flow within 
groups (e.g., Bales, 1950) and involved classifying each speech act as to type (i.e., 
observations regarding flight status, inquiries seeking information, etc.). The findings 
were clear: crews who communicated more overall tended to perform better and, in 
particular, those who exchanged more information about flight status committed fewer 
errors in the handling of engines and hydraulic and fuel systems and the reading and 
setting of instruments. 

This methodology has been subsequently refined by Barbara Kanki and her 
colleagues at NASA—Ames Research Center and applied to communications records 
from additional experimental simulations. Kanki, Lozito, & Foushee (1989) and Kanki 
& Foushee (1989) examined communications patterns among crews in the previously 
described fatigue simulation (Foushee et al., 1986). For example, in the Kanki et al. 
study, sequences of communications were classified in terms of initiator and target as well 
as content. Initiating communications were classified as commands, questions, observations, 
and dysfluencies (e.g., ungram-matical or incomplete statements), while responses were 
classified as replies (responses greater than simple acknowledgments), acknowledgments, or 
zero response. Over and above the typical (and prescribed) occurrences of command— 
acknowledgment sequences, this study found that greater information transfer in the 
form of “commands” structuring activities and acknowledgments validating actions was 
associated with more effective crew performance. 

Communications sequences were contrasted between crews committing a large 
number of operational errors and those making few. Although some specific patterns 
(such as that noted above) are worth special note, the primary finding of the study was 
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the homogeneity of patterns characterizing the low-error crews. This was interpreted as 
the adoption of a more standard, hence more predictable form of communication. 
High-error crews, in contrast, showed a great diversity of speech patterns. Kanki and 
Palmer further discuss the status of communications research as it relates to flightcrews in 
their chapter. 

Orasanu (1991) has conducted additional analyses of decision-making by crews in 
this simulation and has identified four components that support the decision process and 
differentiate effective from ineffective crews. This decision strategy includes situation 
assessment, metacognitive processes in forming action plans, shared mental models based on 
intra-crew communication of both situation assessment and plans, and resource 
management that encompasses task prioritization and delegation of specific 
responsibilities. Orasanu’s formulation is congruent with basic principles of CRM and 
can be translated into prescriptive training. Several airlines have incorporated these 
findings and concepts into their CRM training. This research and a growing empirical 
and theoretical literature question traditional theories of decision making that are based 
on the assumption of a “rational,” but biased, Bayesian decision maker (e.g., Klein, 
Orasanu, Calder-wood, & Zsambok, in press). In particular, this approach emphasizes 
differences between decision-making by experts in natural settings with high stakes and 
time pressure, and the processes employed by naive subjects in the constrained, 
laboratory environments frequently employed in decision research. Orasanu summarizes 
the state of knowledge in this area in her chapter. 

Data from the Chidester et al. (1990) simulation involving personality factors were 
coded and analyzed to isolate decision-making processes while crews dealt with 
multiple inflight abnormalities—a jammed stabilizer and low oil pressure on one 
engine (Mosier, 1991). It was found that the majority of crews utilized a strategy 
consistent with Thordsen & Klein’s (1989) team decision model. Sampling of 
information and repeated verification of the accuracy of situation assessment 
continued throughout the decision process. Many crews made preliminary, revocable 
decisions as soon as they felt they had enough critical data about the problem. The 
implication of this finding is that, while thorough assessment of the situation is 
critical, crews make decisions without having all relevant information. Indeed, the 
best-performing crews collected information pertinent to situation evaluation after 
making a final decision as a means of confirming the decision. In contrast, high-error 
crews showed a diverse pattern of interactions. 

In a field investigation of group formation and interaction processes among 
three-person airline crews, Ginnett (1987) observed crews from their formation on the 
ground prior to the first flight of a multi-day trip, and in the cockpit on each flight 
segment. He found that the quality of the initial briefing was associated with better crew 
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performance throughout the trip. Captains of effective crews communicated the 

team concept and elaborated or affirmed the rules, norms, and task boundaries that 
constitute the organizational structure (what Hackman, 1987, has called the 
“organizational shell’) in this first encounter. Leaders of less effective crews showed a 
variety of interaction patterns. Thus in both studies there was consistency among 
crews rated as performing well and diversity among the less effective teams. These team 
issues are discussed in the chapter by Ginnett. 


1.4.5. Elaborating Group Process Factors 


Building on research with flightcrews and theoretical conceptions of group process 
mediators of aircrew performance, we should be able to fill in the black box with 
a more complete description of the processes that influence outcomes. Helmreich, 
Wilhelm, Kello, Taggart, & Butler (1991) have developed an evaluation system for 
systematic observation of flightcrews in line operations and simulations. The 
methodology grew out of findings from small group research and investigations of 
accidents and incidents. Group processes identified during flight operations fall into 
two broad categories. One consists of the interpersonal and cognitive functions. The 
second includes machine interface tasks. The latter category reflects the technical 
proficiency of the crew. It is a given that optimal team interactions and decision- 
making will be of little value if the crew cannot also integrate them with technical 
execution of maneuvers and procedures needed for safe flight. There is also ample 
evidence from review of the accidents cited earlier that competence in machine 
interface tasks alone does not guarantee operational safety. 

Figure 1.4 shows the expanded group process model as it flows into outcome factors. 
In theory, the two categories of group processes containing human factors and technical 
components must be integrated operationally to produce effective overall performance. 
Note that the final box in Figure 1.4 is labeled “Integrated CRM and Technical 
Functions” to emphasize the fact that the two components need to come together in the 
group process phase, which then flows into desired outcomes of safe and efficient 
mission completion. 

Breaking the subordinate categories down further, die interpersonal and cognitive 
functions can be classified into three broad clusters of observable behaviors: team 
formation and management tasks, communications processes and decision tasks, and 
workload management and situation awareness tasks. The machine interface tasks fall 
into two clusters, the actual control of the aircraft (either manually or through 
computer-based flight management systems) and adherence to established procedures 
for the conduct of flight. 


22 Chapter 1 • Why CRM? Empirical and Theoretical Bases of Human Factors Training 


Figure 1.4 Flightcrew performance model: Expanded group process factors 
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The first cluster deals with the formation of the crew as an operating team, including 
cabin as well as flightdeck personnel. As Ginnett’s (1987) research has demonstrated, 
there is a formation process for teams during which patterns of communication and 
interaction are established. Once established, the process continues and leads to activities 
that can maintain patterns of effective (or ineffective) group interaction. The process of 
formation and maintenance can be categorized into two broad areas, leadership, 
followership, and task concern; and interpersonal relationships and group climate. 

Flightcrews are teams with a designated leader and clear lines of authority and 
responsibility. Not surprisingly, the captain, as leader, can and should set the tone of the 
group. Effective leaders use their authority but do not operate without the participation 
of other team members. As demonstrated in the Chidester et al. (1990) simulation study, 
captains’ attributes such as personality play a role in determining group processes and 
outcomes. Two negative patterns of leadership have been isolated in the investigation of 
accidents. One consists of a strong, autocratic leader who chills input from subordinates 
and conducts operations as if the flightdeck were a single-seat fighter. The “macho 
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pilot” tradition discussed by Foushee & Helmreich (1988) represents the prototype of 
such a leadership style and is typified by an incident reported by Foushee (1982) in 
which a co-pilot’s attempts to communicate an air traffic control speed restriction were 
met with an order to “just look out the damn window.” Equally destructive are leaders 
who abdicate responsibility and fail to control activities on the flightdeck. An example of 
this type of leadership is seen in the crash of a B-727 at Dallas—Fort Worth because the 
crew was distracted and failed to confirm that flaps were set prior to take-off (NTSB, 
1989). In this case, the first officer became involved in a lengthy social conversation with 
a flight attendant during taxi. Although not participating extensively in the 
conversation, the captain failed to control the group processes and did not establish work 
priorities or demonstrate a concern for operational duties. 

One of the observable components of group processes is the quality of interpersonal 
relationships and the resulting group climate. Effective crews maintain a group climate 
that encourages participation and exchange of information. The group climate does not 
reflect the crew’s concern with effective accomplishment of required tasks, but it is 
axiomatic that, other things being equal, crews functioning in a positive environment 
will be more motivated and will participate more fully in team activities. 


Communications processes and decision tasks 


As data from experimental simulations have shown, the processes of information transfer 
and decision-making are prime determinants of crew performance, and higher levels of 
communication are associated with fewer operational errors. Critical elements in this 
process include briefings and the extent to which free and open communications are 
established and practiced. Briefings need to address team formation issues as well as 
technical issues anticipated during operations. Although categorized as part of the 
communications cluster, briefings are one of the demonstrated means of forming 
effective teams and establishing a positive group climate. 

Inquiry, advocacy, and assertion define behaviors meant to ensure that necessary 
information is available and that required communications are completed at 
appropriate times (for example, initiating and completing checklists, alerting others to 
developing problems). The accident literature is replete with examples of 
crewmembers failing to inquire about actions being taken by others. It is critical to 
safety and team action that crewmembers request clarification when they are unclear 
about the current operational situation or planned actions. Paralleling the need to gain 
operational data is the willingness of crewmembers to advocate effectively courses of 
action that they feel essential to safe and efficient operations. In cases such as the Air 
Florida crash in Washington, D.C. (NTSB, 1982). The voice recorder shows that one 
crewmember is uneasy about the takeoff but fails to express his concern strongly and 
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to advocate an alternative action strategy. Concerns and suggestions for needed actions 
must be communicated with sufficient assertiveness to ensure that others are aware of 
their importance. It is noteworthy that the NTSB’s first call for something like CRM 
was in the form of a recommendation for “assertiveness training” for junior 
crewmembers after investigation of a crash that was caused by fuel exhaustion during 
a hold to investigate a warning light (NTSB, 1979). In this accident, the second officer 
repeatedly reported that the fuel state was critical, but without sufficient assertiveness 
to elicit action on the part of the captain. The willingness of crewmembers to 
advocate the course of action they feel best, even when it involves disagreements with 
others, is an essential attribute of an effective team. When crewmembers have 
differing views of proper courses of action and advocate their preferred course of 
action, interpersonal conflict may result. The observable behaviors resulting from 
disagreement are the means used for conflict resolution. Conflict may result in either 
careful consideration of alternatives, or a polarization of positions and a negative 
group atmosphere. Effective conflict resolution is focused on what is right rather than 
who is right. 

Active participation in decision-making processes should be encouraged and 
practiced, including questioning actions and decisions. When decisions are made, they 
need to be clearly communicated and acknowledged. Crew self-critique is another 
essential component of effective group processes. Teams need to review their decisions 
and actions with the goal of optimizing future team activities. Effective critique includes 
the product or outcome, the process, and the people involved. Critique can and should occur 
both during and after completion of activities. Critique is not the same as criticism. 
Indeed, review of effective team performance is a powerful reinforcer. 


Situation awareness, workload management tasks 


The third grouping of crew effectiveness markers is labeled Workload management and 
situation awareness. The crew’s awareness of operational conditions and contingencies, 
usually defined as situation awareness, has been implicated as causal in a number of 
incidents and accidents. However, situation awareness is an outcome rather than a specific 
set of mission management behaviors. The specific factors that are defined for this cluster 
are preparation/planning/vigilance, workload distribution, and distraction avoidance. 

Preparation, planning, and vigilance behaviors reflect the extent to which crews 
anticipate contingencies and actions that may be required. Excellent crews are always 
ahead of the curve while poor crews continually play catch-up. Vigilant crews devote 
appropriate attention to required tasks and respond immediately to new information. 
However, a crew indulging in casual social conversation during periods of low workload 
is not lacking in vigilance if flight duties are being discharged properly and the 
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operational environment is being monitored; the crew may be using this time for team 
formation and maintenance. 

As the Ruffell Smith (1979) study demonstrated clearly, when abnormal situations 
arise during a flight, particular crewmembers may become overloaded with multiple 
tasks and/or become distracted from primary responsibilities. One of the observables of 
group process is how well crews manage to distribute tasks and avoid overloading 
individuals. By prioritizing activities, teams can avoid becoming distracted from essential 
activities, as was the crew whose concentration on a burned-out light bulb kept them 
from noticing that the autopilot had become disengaged and that the aircraft was 
descending below the proper flight path (NTSB, 1972). 


Machine interface tasks 


The flight control and procedural tasks that constitute the machine interface portion of 
group processes represent the traditional model of flight training and evaluation. The 
model proposed here, with its inclusion of interpersonal and cognitive processes, in no 
way downplays the continuing importance of these activities. Rather it reflects the fact 
that both are essential to safe and efficient operations. 

If the proposed model does indeed reflect the major input and process determinants 
of flightcrew performance, it should provide insights into how training programs can 
best address the group processes of flight. In the following section we discuss theoretical 
approaches to maximizing the impact of CRM. 


1.5. Theoretical Leveraging of CRM Training 


The model indicates that there are multiple determinants of crew effectiveness among 
both input and process factors. In theory, organizations should achieve the greatest 
impact on crew performance when they address and optimize as many input and group 
process factors as possible. In this section we consider how programs can be designed to 
accomplish this. This discussion is cast in terms of an integrated approach to technical 
and human factors training. 


1.5.1. Optimizing Input Factors 


Individual factors 


We suggested in an earlier article on crew interaction and performance that the 
selection of individuals more predisposed toward team activity and crew coordination 
concepts could provide one means of achieving more effective crew Performance 
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(Foushee & Helmreich, 1988). Subsequent research has supported this contention as 
personality factors have been linked to crew performance in experimental simulations 
(Chidester et al., 1990), to acceptance of CRM training and changes in attitudes 
regarding flightdeck management (Chidester et al., 1991; Helmreich & Wilhelm, 1989, 
1991; Helmreich, Wilhelm, & Jones, 1991), and to fatigue and health complaints in 
short- and long-haul operations (Chidester, 1990). The chapters by both Hackman and 
Chidester discuss the need for innovations in this area. Selection represents a long-term 
strategy, but one that should be entertained. In the short term, however, efforts should 
concentrate on enhancing training for the existing workforce. 

All effective training programs have an information base. In the case of CRM, the 
goal is to communicate new knowledge about effective team performance and, 
concurrently, to change or reinforce attitudes regarding appropriate flightdeck 
management. Changed attitudes, in turn, should be reflected in improvements in group 
process and ultimately in better crew performance. 


Organizational factors 


There are a number of issues that organizations can address that should, in theory, 
increase crew effectiveness. Foremost, of course, is to demonstrate a commitment to 
developing and implementing training of the highest quality. However, unless the 
concepts presented in training are consistent with the organization’s culture and 
practices, they are not likely to have a major impact. Several steps are necessary to ensure 
that the culture and norms are congruent with CRM. One is to stress training using 
a crew rather than an individualistic model. Another is to make checklists and other 
cockpit documents consistent with crew concepts (Pan American Airways took this step 
in the early 1970s in response to a number of crew-induced accidents). An additional 
step is to address communications issues between flightcrews and other operational units 
including dispatchers, cabin crews, and the maintenance force. The interface between 
the cockpit and these elements forms a significant component of group processes and 
can either support or hinder effective team performance. 

An essential means of making organizational culture and norms congruent with 
CRM concepts is by providing role models who practice and reinforce them. In most 
organizations, check airmen, instructors, and chief pilots are highly respected and 
experienced pilots who are looked to as exemplars of the organization’s norms and 
requirements (Helmreich, 1991a, 1991b; Helmreich, Wilhelm, Kello, Taggart, & 
Butler, 1991). Selection of individuals for these positions should include assessment of 
interpersonal as well as technical expertise. Special training in evaluating and debriefing 
group processes can help them establish and maintain norms supportive of good CRM 
practices. 
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Regulatory factors 


In 1986, following a crash caused by a crew’s failure to complete pre-take-off checklists 
and to extend flaps, then FAA Administrator T. Allen McArtor called a meeting of 
airline managers to discuss the implementation of human factors training. This resulted 
in the formation of a government—industry working group that drafted an Advisory 
Circular (AC) on cockpit resource management (FAA, 1989; in press). The AC defines 
the concept, suggests curriculum topics, and recognizes that initial CRM training 
provides only basic awareness of CRM issues. It further points out that awareness must 
be followed by a practice and feedback phase and a continual reinforcement phase. Full 
mission simulation training (line-oriented flight training, LOFT) is highly 
recommended as the most effective means of continual reinforcement. The content of 
the AC is consistent with generally accepted principles of learning and reinforcement 
and with the theoretical model of flightcrew performance being discussed here. 
Although CRM has not been mandated as a requirement for air carriers, the AC clearly 
encourages U.S. carriers to develop such programs. Efforts are further under way to 
mandate CRM training for all air transport. 

Also growing out of this government—industry collaboration has been a Special 
Federal Aviation Regulation—Advanced Qualification Program (FAA SFAR 58, AQP) 
issued in 1990. AQP is described in detail in the chapter by Birnbach & Longridge. It is 
a voluntary regulation for airlines that allows much more flexibility and innovation in 
training. In exchange for this flexibility in conducting training, participating airlines are 
required to provide CRM training, LOFT, and to initiate formal evaluation of crew as 
well as individual proficiency. Organizations that operate under AQP should find the 
regulatory environment supportive of CRM training efforts. 


1.5.2. Enhancing Group Process Factors 


In theory, the point of greatest impact on flightcrew behavior should be the group 
process itself. This should be accomplished effectively by full mission simulation training 
(LOFT), where crews have an opportunity to experiment with new interaction 
strategies and to receive feedback and reinforcement. The FAA supported this approach 
and issued an Advisory Circular (FAA, 1978) establishing guidelines for the conduct 
of LOFT. NASA hosted an industry conference on LOFT in 1981 that resulted in two 
volumes providing a review of techniques and formal guidelines for its conduct 
(Lauber & Foushee, 1981). The principles espoused include establishing high levels 
of realism, conducting normal flight operations as well as creating emergency and 
abnormal situations, and nonintervention by instructors into group processes, decisions, 
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and actions. CRM LOFT is defined as training rather than formal evaluation, with the 
goal of allowing crews to explore the impact of new behaviors without jeopardizing 
their certification as crewmembers. 

LOFT should influence subsequent behavior most strongly when scenarios are 
crafted to require team decision-making and coordinated actions to resolve in-flight 
situations. The debriefing of LOFT is also a critical element in achieving impact. Skilled 
instructors should guide crews to self-realization rather than lecture them on observed 
deficiencies. Instances of effective team behavior should be strongly reinforced. The 
use of videotapes of the simulation can provide crews with the opportunity to examine 
their own behavior with the detachment of observers (Helmreich, 1987). Butler 
discusses the status of contemporary LOFT programs in his chapter and Wiener discusses 
the peculiarities of LOFT in the high-technology cockpit in his chapter. 

In addition to the practice and reinforcement provided later by LOFT, initial CRM 
training, usually conducted in a seminar setting, should allow participants to observe and 
experiment with behavioral strategies and to receive individual and group feedback. 
Instruction that allows participants to experience processes is more meaningful than 
lectures where ideas are presented to a passive audience. Introductory training in CRM 
provides the conceptual framework needed to understand the processes that will later be 
encountered in LOFT. 

It is also necessary to identify and reinforce effective group processes in normal line 
operations as well as in the training environment. We earlier identified check airmen as 
key agents and role models. To help transfer concepts from training to the line, check 
airmen should address not only technical performance but also interpersonal and 
cognitive issues in their conduct of periodic evaluations of crew performance line 
operations (line checks). 

As we pointed out in describing Figure 1.4, process factors from both the 
interpersonal and machine interface components need to be integrated as the team 
performs its duties. The corollary of this is that the most effective training should bring 
together technical and human factors aspects of each maneuver taught, so crewmembers 
can recognize that every technical activity has team-level components essential to its 
successful completion. For example, the V; cut? is a maneuver in which crews are 
required to demonstrate proficiency. It involves the loss of power at a point when it is too 
late to abort the take-off. Crews are required to climb out, reconfigure the aircraft, 





3V} is the decision speed for take-off. When an aircraft reaches V, the crew is 
committed to take-off. It is a function of runway length and condition, aircraft 
weight, temperature, etc. We are indebted to Captain Kevin Smith for his analysis of 
actions required during the maneuver. 
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communicate with the tower, and return for landing. While this is often seen as 
primarily a technical exercise, in fact it requires concerted activity by the full crew along 
with rapid, accurate information transfer within the cockpit and between cockpit and 
cabin and cockpit and ground. If training in basic flight maneuvers stresses the human 
factors as well as technical components, the likelihood that crews will demonstrate 
effective, integrated group processes should be increased. 

In a similar vein, the specificity of concepts communicated and reinforced should 
determine their acceptance and adoption. Individuals may accept, in principle, abstract 
ideas of open and complete communication, team formation, situation awareness, and 
workload management, but may find it difficult to translate them into concrete 
behaviors on the flightdeck. In theory, individuals who understand both the conceptual 
bases of effective crew coordination and their specific behavioral manifestations 
should be able to put them into practice readily and should be able to evaluate their 
success in accomplishing them. 

As part of a research effort to evaluate the impact of CRM training and to train 
observers to judge crew effectiveness, Helmreich, Wilhelm, Kello, Taggart, and Butler 
(1991) have attempted to define behavioral markers of the three clusters of interpersonal 
and cognitive tasks. These are observable behaviors that reflect the concepts central to 
CRM training. Forty discrete markers have been isolated and utilized in observations of 
line operations and LOFT (Clothier, 1991a). The data suggest that these behaviors can 
be reliably measured. Figure 1.5 shows the ten markers associated with the Situation 
Awareness/ Workload Management cluster. It can be argued that programs that employ 
concrete, behavioral examples should have a greater impact on crew processes and 
outcomes than those that deal with abstract concepts. 


Figure 1.5 Behavioral markers for workload distribution/situational awareness 


Avoids “tunnel vision”, being aware of factors such as stress that can reduce 
vigilance 

Actively monitors weather, aircraft systems, instruments, and ATC, sharing 
relevant information 

Stays “ahead of curve” in preparing for expected or contingency situations 
Verbally insures that cockpit and cabin crew are aware of plans 

Workload distribution is clearly communicated and acknowledged 


Ensures that secondary operational tasks are prioritized 

Recognizes and reports work overloads in self and others 

Plans for sufficient time prior ro maneuvers for programming of automation 
Ensures that all crewmembers are aware of status and changes in 
automation 

Recognizes potential distractions caused by automation and takes 
appropriate preventive action 
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In this section we have tried to derive approaches to CRM training that should 
theoretically have the greatest leverage on crew performance. This analysis suggests that 
programs need to attack a number of areas in concert if they are to achieve maximum 
influence on behaviors and attitudes. In the following section we discuss efforts to 
achieve these goals and describe some of the major developments in CRM training over 
the last decade. 


1.6. The Evolution of CRM Training 


Formal training in human factors aspects of crew operations was beginning to take root 
by the 1970s. For example, the late Frank Hawkins (1984) had initiated a human factors 
training program at KLM, Royal Dutch Airlines, based on Edwards’ (1972, 1975) SHEL 
model and trans-cockpit authority gradient. Operational and theoretical concerns with 
human factors aspects of flight came together in a NASA/Industry workshop held in 
1979. At this gathering, managers from worldwide aviation met with the members of 
the academic and government research community concerned with human 
performance. Research into the human factors aspects of accidents was reviewed (e.g., 
Cooper, White, & Lauber, 1980) along with the seminal findings from the Ruffell Smith 
(1979) study. Many of the participants left the meeting committed to developing formal 
training in crew coordination. 

A number of different CRM courses began to emerge in the early 1980s. The focus 
of most early training was on input factors, especially in the areas of knowledge and 
attitudes. Much of the emphasis was on the review of human factors aspects of accidents, 
with the goal of changing attitudes regarding appropriate flightdeck management. Many 
of these courses were presented in a lecture format, and some consisted only of 
videotaped presentations. Other training, growing out of management development 
programs, included tests and exercises designed to provide self-awareness and to 
demonstrate general concepts of group processes. What was not present in early efforts 
was a focus on organizational issues and flightcrew group processes, including 
reinforcement of effective process behavior. Many early CRM courses faced 
considerable resistance from crewmembers who expressed concerns about both the 
motivation for and possible outcomes of the training. Some saw it as unwarranted 
psychological meddling, equating the training with clinical psychology or 
psychotherapy. Others feared that captains’ authority would be eroded by a kind of Dale 
Carnegie charm school approach to developing harmonious interpersonal relations, 
without regard for operational effectiveness. 

The first CRM course integrated with LOFT was developed by United Airlines 
following the NASA workshop. The course, called Command, Leadership, and 
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Resource Management, was the result of a collaboration among United flight training 
personnel, members of the Air Line Pilots’ Association, and Drs. Robert Blake and 
Jane Mouton. Blake and Mouton were social psychologists who had developed 
training programs aimed at improving managerial effectiveness for a number of major 
corporations. The centerpiece of their training approach is providing participants with 
insights into their personal managerial styles (an individual input factor) using the 
managerial grid (Blake and Mouton, 1964) as a means of classifying managers along 
independent dimensions of task and interpersonal orientations. The multi-day training 
program that emerged is intensive and interactive, requiring participants to assess their 
own behaviors and those of peers. Operational concepts stressed in the training 
include process factors such as inquiry, seeking of relevant operational information; 
advocacy, communicating proposed actions; and conflict resolution, decision-making, 
and critique, reviewing actions taken and decisions reached. The unique aspect of the 
United approach was that the initial training was followed by recurrent review of 
CRM concepts. The program also demonstrated a major commitment to group 
process factors by providing annual CRM LOFT sessions. These allow crews to 
practice the human factors concepts covered in the seminar and recurrent training. 
One of the major innovations in United’s LOFT was the use of a video camera in the 
simulator to record crew interactions. By replaying the tape of their LOFT, crews gain 
the ability to review their actions and decisions and to obtain insights into their 
behavior, guided by the LOFT instructor.* This program represents the first 
integration of multiple input and group process factors that also recognized the need 
for continuing practice and reinforcement. 

NASA and the Military Airlift Command of the U.S. Air Force jointly sponsored 
a workshop on developments in CRM training in May, 1986 (Orlady & Foushee, 1987). 
This conference demonstrated the striking spread of CRM training throughout the 
world since the first workshop in 1979. Reports were presented on the implementation 
of CRM courses at United Airlines (Carroll & Taggart, 1987), Pan American World 
Airways (Butler, 1987), People Express Airlines (Bruce & Jensen, 1987), Continental 
Airlines (Christian & Morgan, 1987), Japan Air Lines (Yamamori, 1987), Trans Australia 
Airlines (Davidson, 1987), in units of the Military Airlift Command (Cavanagh & 
Williams, 1987: Halliday, Biegelski, & Inzana, 1987), and in corporate and regional 
operations (Mudge, 1987; Schwartz, 1987; Yocum & Monan, 1987). 

In the late 1980s a second generation of CRM training began to emerge in the 
United States. Pan American World Airways and Delta Airlines both initiated CRM 





*The videotape is always erased following the LOFT debriefing to preserve the 
confidentiality of the training and behaviors observed. 
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courses that included recurrent classroom training and LOFT. In addition, these 
programs addressed organizational input factors by providing additional training for 
check airmen and instructors with the goal of increasing impact on group process 
factors through reinforcement of effective behaviors both in LOFT and in line 
operations. Black and Byrnes discuss the implementation of the Delta program in 
their chapter. 

Although there has been a great proliferation of CRM courses, there has not been 
a parallel growth in the use of CRM/LOFT to provide practice and reinforcement. At 
the time this is written, in the United States only United, Horizon Airlines, Delta, 
Continental, and units of military aviation have integrated CRM/LOFT programs, 
although a number of other organizations including Northwest Airlines, US Air, and 
Comair are in the process of implementing them. There are a number of reasons why 
more comprehensive programs have been slow in emerging. One is certainly economic. 
As Chidester points out in his chapter, at a time of great financial distress in the industry, 
innovative and relatively expensive programs that are not formally mandated by 
regulations must compete with other operational needs for scarce resources. Indeed, 
regulations in the U.S. have tended to operate against the adoption of LOFT because it is 
necessary to meet many formal, technical requirements each year and because 
requirements for recurrent training for captains are semi-annual but annual for first 
officers and flight engineers, making it difficult to schedule complete crews for LOFT.” 
The previously mentioned Advanced Qualification Program both removes some of the 
regulatory barriers to comprehensive CRM/LOFT and provides incentives for their 
adoption. Additional resistance to changes in training may also come from awareness that 
the aviation system has an excellent safety record when compared with all other forms of 
transportation and from the fact that empirical evidence for increased safety of flight as 
a result of CRM training has been lacking until very recently. 

At the present time a third generation of CRM training is emerging. This approach 
continues the practices of integrating CRM with LOFT but also takes a systems 
approach to multiple input factors including organizational cultures and group and 
individual factors. Evaluation and reinforcement in line operations are also cornerstones 
of this approach. In addition, new programs are becoming more specific in focus and are 
defining and directly addressing optimal behaviors (e.g., behavioral markers). Efforts are 
underway in several organizations (stimulated in part by requirements of AQP) to 
remove the distinction between technical training and evaluation and CRM, with the 





° United Airlines, Pan American Airlines, and Delta Airlines have received exemptions 
from some training requirements to facilitate training complete crews on an annual 
basis in exchange for implementation of integrated CRM/LOFT programs. 
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goal of implementing a training philosophy where both components are addressed in 
every aspect of pilot qualification. 

An additional characteristic of evolving programs is the extension of CRM training 
beyond the cockpit to other operational areas. Joint training for cabin and cockpit crews 
has been initiated at America West Airlines, and programs are being developed at 
a number of other carriers. American Airlines is including dispatchers in CRM training 
in recognition of common concerns and responsibilities and the need for effective, open 
communication. Pan American and later Continental Airlines developed CRM 
programs for maintenance personnel. Efforts are also underway to implement similar 
training within the FAA for Air Traffic Control personnel who also operate in a team 
environment but have historically received little or no formal instruction in human 
factors issues relating to their jobs. 

Looking at the growth and evolution of CRM training, one is struck by the 
willingness of very disparate organizations to embrace a training concept that counters 
many of the traditions of an industry. In the following section we consider factors that 
may have facilitated this acceptance. 


1.7. CRM and Traditional Management 
Development Training 


From an observer’s perspective, the philosophical and pragmatic bases of CRM are 
consistent with programs that have been used in management development training for 
several decades. Concerns with self-assessment, managerial styles, interpersonal 
communications, and organizational influences on behavior have academic roots in 
social, industrial, and clinical psychology, sociology, and schools of business. Programs 
to translate empirical and theoretical knowledge about groups into practical training 
have been employed with differential acceptance in many segments of industry and 
government. Indeed, many of the initial CRM programs, such as that at United Airlines, 
were adaptations of existing management training courses. What is striking about CRM 
is the rapidity of its spread and the enthusiasm with which it has been accepted. What is 
unique about its implementation in this setting? What can convince fiscally conservative 
managers to commit scarce resources and highly experienced crewmembers to 
re-evaluate their approach to a highly structured task? 

Part of the answer rests in the nature of the flight environment. Operating an aircraft 
with a multi-person crew is a structured and bounded endeavor with clear lines of 
authority and responsibility. The inherent activities involved in taking an aircraft from 
one point to another are similar in organizations throughout the world. Although 
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aircraft differ in design and sophistication and in number of crewmembers required for 
operation, the basic tasks are generic. One implication of this is that the types of 
problems in flightdeck management found in one organization or flightcrew have a high 
probability of occurring in others. Findings regarding crew contributions to accidents 
can be easily recognized as generic rather than as unique occurrences in unique 
organizational cultures and operating environments. It can be inferred that similar 
approaches to improving crew effectiveness should work throughout the industry 
despite differences in the culture, history and health of organizations. The chapters by 
Johnston; Yamamori & Mito: and Helmreich, Wiener, & Kanki provide additional 
perspectives on cross-national issues in human factors. 

In aviation the results of breakdowns in flightcrew group processes are dramatic and 
highly visible and provide an unequivocal outcome criterion. In contrast, outcome 
criteria in industry such as profits or productivity are relatively diffuse and subject to 
qualification by industry-specific and organization-specific factors. Given an overall 
performance criterion that represents a common, desired outcome, it is understandable 
that a similar approach would be recognized and embraced. 

Again, in contrast to the diversity found outside aviation, the range of decisions 
and behaviors that faces flightcrews is constrained and can be incorporated in a fairly 
simple model. Because of this behavioral specificity, training can be more sharply 
focused than it normally is in courses developed for generic managers. This clearer 
definition of issues and processes should lead both to greater acceptance by participants 
and to more tangible, positive outcomes. 

Another distinctive feature of the aviation environment is the ability to use highly 
realistic simulation to practice behaviors and receive feedback and reinforcement. 
Unlike many of the exercises that are used in general management training, LOFT 
provides a valid representation of the actual task setting with measurable outcomes. This 
allows crews to observe the discrete components of group processes as they flow into 
outcomes. LOFT provides compelling evidence of the validity of the concepts being 
trained. 

The ultimate question, of course, is how well the training achieves its stated goals. In 
the following section we review preliminary results from evaluation of CRM courses in 
a number of organizations. 


1.8. Research Findings 


Although the process of research is necessarily slow and incremental, a number of 
consistent findings have emerged regarding the effects of CRM programs. Our goal is to 
provide a brief overview of what research has told us about the impact of CRM and to 
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point out some of the gaps in current knowledge. It should be noted that the research to 
be discussed regarding the effectiveness of CRM training comes from evaluation of 
intensive programs integrated with LOFT and not from brief lecture or discussion 
sessions called CRM that may be included in crew training. Strategies for the 
investigation of CRM-related behaviors and concepts are discussed further in 
Helmreich (1991b). 


1. Crewmembers find CRM and LOFT to be highly effective training. Survey data from 
more than 20,000 flight crewmembers in civilian and military organizations in 
the United States and abroad show overwhelming acceptance of the training. 
The vast majority of crewmembers find the training both relevant and useful 
(Helmreich & Wilhelm, 1991). Figure 1.6 shows the distribution of responses in 
five airlines to a post-training survey question regarding the utility of the 
training. 

A similar pattern of endorsement is found in evaluations of the value of 
LOFT. Wilhelm (1991) has analyzed reactions to LOFT from more than 8,000 
participants in the training at four organizations. Crewmembers overwhelm- 
ingly feel that it is important and useful training and that it has value on the 
technical as well as the human factors dimensions. Figure 1.7 shows the distri- 
bution of mean ratings of the usefulness of LOFT in four airlines, broken down 
by crew position. 

Clearly, acceptance of training is a necessary but not sufficient indicator of its 
effectiveness. If crews do not perceive training as useful, it is unlikely that it will 


Figure 1.6 Responses to the question, “Overall, how useful did you find the CRM 
training?” in five organizations (A, B, C, D, E) 
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Figure 1.7 Average ratings for the item, “Overall, LOFT is an extremely useful 
training technique,” in four organizations (A, B, C, D). Scale: 1, strongly disagree: 
4, neutral: 7, strongly agree 
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induce behavioral change. On the other hand, the training may be perceived 
as useful, but because behavioral tools are not provided to help participants 
apply the concepts, the result may be increased awareness of CRM concepts but 
little change in observable behavior. 


2. There are measurable, positive changes in attitudes and behavior following the introduction 
of CRM and LOFT, Changes in attitudes regarding flightdeck management 
measured by the CMAQ (Helmreich, 1984) can be used as a measure of training 
impact. Typically, attitudes show significant positive shifts on the three scales of 
the CMAQ, Communications and Coordination, Command Responsibility, 
and Recognition of Stressor Effects (Helmreich & Wilhelm, 1991). As 
Figure 1.8 illustrates for the Communications and Coordination scale in six 


Figure 1.8 Pre-test (unshaded) and post-training (shaded) attitudes on the CMAQ 

Communications and Coordination scale. All differences significant (p < .01): scale 
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organizations, there is a consistent increase in the positivity of reactions, 
although the magnitude of change (along with the baseline attitudes) varies 
between organizations. The CMAQ findings suggest that participants do relate 
the concepts being taught to specific attitudes regarding the conduct of flight 
operations. 

Because the linkage between attitudes and behavior is less than perfect (e.g., 
Abelson, 1972), it is critical to the validation of CRM training effectiveness that 
there be observable changes in crewmembers’ behaviors on the flightdeck. Data 
have been gathered both by independent observers and by check airmen and 
instructors given special training in observational methodology (e.g., Clothier, 
1991b). Data collected across time show changes in behavior in the desired 
direction. Figure 1.9, for example, shows shifts in observed behavior during line 
operations over a 3-year period on 14 observed categories of process behavior 
following the introduction of CRM and LOFT in one major airline. All mean 
differences are statistically significant. It can be noted that the behavioral effects 
continue to grow across time. A reasonable interpretation of this trend is that, as 
concepts become more widely accepted, organizational norms shift and exert 
pressure on crewmembers to conform to the new standards of behavior. 


Figure 1.9 Average crew performance ratings in one organization across time. 
Scale: 1, poor; 5, excellent 
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Significant differences have also been found when crew behavior is aggre- 
gated and contrasted in terms of the level of flightdeck automation (Butler, 
1991: Clothier, 1991a). Crews observed in advanced technology aircraft are 
rated as more effective in LOFT than those flying conventional aircraft on 
a number of human factors dimensions. The causes and extent of these differ- 
ences remain for further research to clarify. Issues surrounding cockpit auto- 
mation, crew coordination, and LOFT are discussed in the chapter by Wiener. 

As we have noted, the number of accidents involving crews with formal 
training in CRM and LOFT is too small to draw any statistical inferences 
regarding the role of these experiences in helping crews cope with serious 
emergency situations. There are, however, a growing number of anecdotal 
reports that the training does provide valuable resources for crews faced with 
major inflight emergencies. Two recent accidents have involved United Airlines 
crews with both CRM and LOFT experience. In one, a cargo door blew off in 
flight on Flight 811, a Boeing 747, causing considerable structural damage and 
the loss of two engines. In the other, the catastrophic failure of the center engine 
on a McDonnell Douglas DC-10, Flight 232, resulted in the loss of all hydraulic 
systems and flight controls. Both crews were able to minimize loss of life by 
coping effectively with the problems, and both acknowledged the role of CRM 
in enabling them to cope with their novel emergencies. Crew communications 
taken from the cockpit voice recorder transcripts have been coded in terms of 
content and frequency and analyzed by Steven Predmore (1991). The coding 
system classifies communications in terms of CRM concepts including inquiry, 
command and advocacy, reply and acknowledgment, and observation 
(communication of operational information). Both crews maintained a high 
level of communication and verification of information throughout the emer- 
gencies. Figure 1.10 shows the pattern of communications over time in both 
accidents. 


3. Management, check airmen, and instructors play a critical role in determining the effec- 
tiveness of CRM training. Hackman’s (1987) delineation of the “organizational 
shell” as a critical determinant of the success of CRM training has been borne 
out by operational experience and research. Organizations where senior 
management has demonstrated a real commitment to the concepts of CRM and 
its importance for safety and crew effectiveness by providing intensive and 
recurrent training have found greater acceptance than those which have simply 
provided a brief introduction to the concepts. Indeed, several organizations in 
which flight operations management made a concerted effort to communicate 
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Figure 1.10 Crew communications, by category, during two United Airlines inflight 
emergencies, (a), Flight 811: (b), Flight 232 
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the nature of CRM training and the organization’s dedication have noted 
significant improvement in cockpit management attitudes even before formal 
training was instituted. 

The pivotal position of check airmen and instructors as primary role models 
and agents of reinforcement has also become increasingly recognized 
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(Helmreich, 1987: Helmreich, Wilhelm, Kello, Taggart, & Butler, 1991). 
Consistent with the theoretical model, the extent to which these key individuals 
endorse, practice, and emphasize CRM concepts both in the training and 
checking environment seems largely to determine program acceptance. 


4. Without reinforcement, the impact of CRM training decays. Data indicate that even 
intensive, initial CRM training constitutes only an awareness phase and intro- 
duction to the concepts, and that continuing reinforcement is essential to 
produce long-term change in human factors practices. Some of the most 
compelling evidence of the need for ongoing emphasis on CRM comes from 
revisiting organizations where well-received initial CRM training has not been 
accompanied by an organizational commitment to continuing the effort 
(Helmreich, 1991a). In one organization, when the CMAQ was re-administered 
more than a year after the completion of initial training, attitudes had reverted to 
near their baseline, pre-CRM levels. In this organization many open-ended 
comments written by respondents expressed concern over the fact that some 
outspoken opponents of CRM concepts continued management styles anti- 
thetical to good human factors practice. In another organization, recurrent 
CRM and LOFT were provided, but management support was weak, there was 
high turnover in training and checking personnel, no formal human factors 
training for new check airmen and instructors, and limited efforts to revise and 
update LOFT scenarios. When attitudes regarding the value of CRM training 
and LOFT were assessed more than two years later, they had become signifi- 
cantly less positive than in the first year. These longitudinal findings have major 
operational significance as they reinforce the notion that organizations desiring 
to maintain the momentum provided by initial CRM training must make 
a formal commitment to provide the resources necessary for continuing training 
and reinforcement. 


5. A small but significant percentage of participants “boomerang” or reject CRM training. 
Although the self-report reactions and attitude change findings discussed 
above show the overall positive impact of initial CRM training, some 
participants fail to see its value and some even show attitude change in a 
direction opposite to that intended. These individuals have been described as 
showing a “boomerang effect” (Helmreich & Wilhelm, 1989). Similarly, some 
crews observed in line operations following initial CRM seminars do not 
practice the concepts espoused in training. The fact that reactions to CRM are 
not uniformly positive does not negate the value of the training, but this 
undesired outcome is reason for some concern. 
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Research has shown that there are multiple determinants of the boomerang 
effect (Helmreich & Wilhelm, 1989). Some resistance to the training is rooted in 
individual personality characteristics. Crewmembers who are lacking in traits 
associated with both achievement motivation and interpersonal skills are initially 
more prone to reject CRM concepts. In addition, the group dynamics of 
particular seminars also appear to influence reactions. The presence of a 
charismatic participant who openly rejects the training can influence the level 
of acceptance by other crewmembers and poses a major challenge to those 
conducting the training. 


1.9. Open Issues for Research 


There are a number of open questions that require sustained research efforts to assist 
CRM training in reaching its full potential. One is to determine the long-term impact 
of the training on crew behavior and system safety. Many of the measures employed to 
evaluate crew performance and attitudes are still under development and require 
refinement through research (see the chapter by Gregorich & Wilhelm). Part of the 
measurement effort has been directed toward the development of consistent 
classification strategies for human factors aspects of aviation incidents and accidents. 
These can generate extremely important research databases, and investigations 
supporting this effort are much needed. 

Chidester describes many of the critical issues facing those trying to develop effective 
CRM programs in his chapter. All of these can be addressed more effectively with 
continuing research into the impact of programs and careful assessment of participant 
reactions. Such data should facilitate continual refinement of programs and will take into 
account changes in the aviation system itself (for example, the development of more 
digital data links between aircraft and Air Traffic Control). 

Another urgent need is to learn how to maximize the role of LOFT in reinforcing 
and extending human factors training. Recent data suggest that there are great 
differences in the perceived value of different scenarios and in the quality of their 
implementation (Wilhelm, 1991). The chapter by Butler discusses critical research issues 
that need to be addressed in LOFT design and execution. 

Several critical topics need much additional research before they can be translated 
into basic CRM training. Research into fundamental aspects of interpersonal 
communications, such as that described in the chapter by Kanki & Palmer, has much to 
offer those developing CRM programs, but the knowledge base remains relatively 
undeveloped. Another critical area is decision-making. As Orasanu points out in her 
chapter, substantial progress has been made toward understanding decision-making in 
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natural situations, but much remains to be done before full operational benefits can be 
gained. In particular, additional research into individual and group decision-making 
under highly stressful conditions (such as high time pressure, fatigue, life stresses, and 
life-threatening emergencies) should have high priority. Indeed, the whole topic of 
psychological stress and its behavioral impact has languished in the research community 
and needs renewed attention. Not until the research base is extended will we be able to 
mount effective programs of stress management and evaluate their operational impact.° 

Given the lack of empirical data on the impact of system automation on crew 
coordination, it is also difficult to specify how best to train crewmembers to interact 
most effectively with “electronic crewmembers.” The chapter by Wiener provides 
a summary of the state of our knowledge about behavioral effects of automation, and 
Byrnes & Black describe the first course attempting to integrate automation issues with 
CRM training in their chapter. Clearly this effort will be enhanced by further research. 

We also need to know whether the boomerang reaction to CRM training is 
transitory or enduring. It is characteristic of human nature to question new and alien 
concepts on first encounter. Some exposed to CRM for the first time may show initial 
hostility to the concepts but may, after time and with peer pressure, later become 
enthusiastic advocates of CRM concepts. Only longitudinal research strategies that 
revisit and reassess individual reactions across time can determine the long-term 
reactions of the “boomerang” group. An associated question is whether different 
training strategies or interventions may be needed to gain acceptance from this subset of 
individuals. 

Human factors concepts and training need to be further integrated with traditional 
technical training. To a considerable extent, CRM has developed outside the boundaries 
of the traditional training and evaluation of technical proficiency. As CRM has matured 
and become a part of organizational cultures, awareness of the fact that there are vital 
human factors components of all aspects of flight training has grown. As the theoretical 
model suggests, the effectiveness of both CRM and technical training should be 
enhanced when trainers stress the human factors components of every aspect of flight. 
Only basic research and operational evaluation can optimize these efforts. In the same 
vein, such research should provide guidance for incorporating human factors training 
into initial pilot training as well as training for experienced crewmembers. 





° A related question is what level of stress needs to be imposed on training to maximize 
the probability that human factors concepts will generalize to operational emergencies. 
See the chapter by Butler for further discussion of this topic with regard to LOFT. 
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1.10. Conclusions 


Recognizing the critical role of human factors in determining the effectiveness of 
technically proficient flightcrews in both normal and emergency situations, the aviation 
community has embraced the concept of CRM training. The spread of CRM programs 
has proceeded faster than the accumulation of knowledge regarding their operational 
impact, reflecting the perceived importance of the issues. However, research findings to 
date suggest that this faith has not been misplaced. Crewmembers value the training, and 
available data suggest that it does have a positive impact on crew behavior and, by 
inference, on the safety of the aviation system. 

The theoretical model of flightcrew group processes suggests that the most effective 
CRM courses will simultaneously address multiple input and group process factors and 
will be developed with awareness of the particular cultures in which they are embedded. 
Impact should also be enhanced when participants are not forced to make large 
generalizations from abstract concepts to their normal work setting, but rather receive 
training that communicates psychological concepts in terms of shared everyday 
experiences and clearly defined behaviors. Successful programs appear to provide not 
only basic psychological concepts, but their translation into operational terms. 

It seems likely that if research and evaluation proceed in tandem with the 
implementation of continuing human factors training, courses of the future will evolve 
continually and make today’s efforts look as antiquated as the Link Trainers of World 
War II. The open exchange of information that has developed surrounding CRM 
training has provided an environment conducive to rapid evolution. 
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1.11. CRM Redux 


Revisiting words written 15 years ago was a chastening experience for me. While 

the superordinate goals of CRM training—safe and efficient flight—are the same, its 
scope and practice have changed dramatically. Developments in CRM training and 
guidance for its delivery are provided in an updated Advisory Circular (120.51) of the 
US Federal Aviation Administration (Federal Aviation Administration, 2004). The 
aviation system has also undergone massive upheaval: a faltering economy has resulted in 
bankruptcies and mergers, airline fleets have been reduced in size, and operations have 
been shifted to more efficient, highly automated aircraft flown by two-person crews. 
Extremely long-haul flights, for example Houston to Tokyo, have also been established. 
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On very long flights a full relief flight crew (captain and first officer) is required, raising 
issues of command and leadership in the event of an in-flight emergency. 

One of the factors we did not recognize in 1993 was the powerful influence of 
national culture on flight crew behaviors and the diverse approaches needed for delivery 
and acceptance of CRM programs in different cultures (Helmreich & Merritt, 1998; 
Merritt & Helmreich, 1996c). Another growing realization has been that CRM is not 
for the cockpit alone. (I must confess that as the first edition was going to press there was 
heated debate among the editors about whether the title of the volume should be Crew 
or Cockpit Resource Management. The three of us, Earl Wiener, Barb Kanki and myself, 
ultimately agreed that it should have been Crew Resource Management. 


1.11.1. Culture 


I observed a wide range of cockpit behaviors from the jumpseat (despite assurances from 
managers and check airmen that pilot behavior was highly standardized in their airline). 
To explore this rather startling finding, I designed and administered a survey of pilot 
attitudes, the Cockpit Management Attitudes Questionnaire (CMAQ: Helmreich, 1984). 
The CMAQ was completed by pilots from a number of countries. It queried them about 
their beliefs regarding appropriate cockpit leadership and management of the flight 
deck. Analyzing the data, I was struck by highly significant differences in response as 
a function of aircraft fleet, pilot background and, especially, national culture. It remained 
for my former student and colleague Ashleigh Merritt to develop a new survey, the 
Flight Management Attitudes Questionnaire based on the CMAQ (FMAQ: Helmreich & 
Merritt, 1998). The FMAQ draws on the multi-dimensional conceptualization of 
culture developed by the Dutch psychologist Geert Hofstede (Hofstede, 2001). The 
FMAQ has been administered to flight crews in more than 30 countries. Examining the 
cross-national data, the most diagnostic of Hofstede’s dimensions has proved to be power 
distance (PD). In high PD cultures it is accepted and expected that leaders behave in an 
autocratic manner and it is unacceptable for co-pilots and other junior crew to question 
the captain’s decisions and acts (Helmreich et al., 2001). Asian and Latin American 
cultures tend to be high in PD with Australia anchoring the egalitarian pole and the US 
falling in an intermediate position. One first officer from a high PD culture said to me, 
“I would rather die than challenge the captain’s actions.” Sadly, this statement has been 
borne out in more than one accident (Helmreich, 1994). 

After administering the FMAQ to pilots from an airline in an extremely high PD 
culture, I presented the survey results through a translator to a meeting of senior 
managers and chief pilots. As always I stressed the importance of the first officer 
speaking up when the situation is deteriorating and the aircraft is standing into danger. 
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I was informed later by a bilingual, expatriate pilot at the meeting that while I was 
talking a senior manager announced to all present that they should disregard 
everything I said. 

In the most egalitarian cultures, however, status inequalities are prevalent. In one 
airline from a very low PD culture, organizational rules require that on overnight stops 
the captain must always have a room on a higher floor than the rest of the crew. 

Even without managerial sabotage, gaining acceptance of CRM concepts that run 
counter to culture is a daunting enterprise—especially in cultures where juniors should 
not question or contradict their seniors. I was astonished and delighted to hear how 
a senior captain, head of the CRM program in one Asian carrier, got the CRM message 
across. His admonition to junior pilots was “Think of yourself as the eldest son in 
a traditional family. Your task is to protect your father from harm. Thus it is essential that 
you speak up and warn him if his actions are leading the flight into danger.” 

Clay Foushee and I described CRM as being in its third generation in our chapter in 
the first edition. In the following 15 years another three generations can be identified 
(Helmreich et al., 1999). The fourth generation stressed the definition of procedures that 
include the behaviors exemplifying effective cockpit resource management. The fifth 
generation, known as error management, was short-lived and unpopular. As one captain 
remarked to me, “I feel insulted being labeled as an “error manager’—it implies that my 
job is to screw up and then correct my mistakes.” 

Under the leadership of Captains Bruce Tesmer and Don Gunther of Continental 
Airlines, a sixth generation of CRM emerged, known as threat and error management or 
TEM. TEM is defined and described in the Line Operations Safety Audit (LOSA) 
Advisory Circular 120.70 of the US Federal Aviation Administration (Federal Aviation 
Administration, 2006). TEM gained immediate acceptance from pilots, managers and 
regulators (Helmreich, 1997). TEM accurately depicts the role of flight crews— piloting 
and navigating the aircraft from point A to point B while coping with threats to safety in 
the system and managing errors originating in the cockpit. External threats include air 
traffic controller errors, severe weather, terrain, and a host of others. The TEM concept 
can be applied in all components of an organization—maintenance, dispatch, ramp 
operations, etc. Threat and error management has also proved to be a valuable 
framework for the analysis of CRM-related behaviors in the investigation of air crashes 
(Helmreich, 1994). 

One of the critical issues facing airlines, given the cost of developing and delivering 
training to highly paid staff who expect to be paid for their participation, was whether 
CRM programs change pilot behavior and increase system safety. After experiencing 
a series of embarrassing incidents (including landing at the wrong airport and shutting 
down the good engine after failure of the other), Delta Airlines developed and 


51 


52 


Chapter 1 » Why CRM? Empirical and Theoretical Bases of Human Factors Training 


conducted an intensive three-day CRM course for all its pilots. The course led to 
significant, positive changes in attitudes about CRM but Delta management wanted to 
know if the training also led pilots to change their behavior in normal operations. The 
University of Texas Human Factors Research Project was asked to determine how well 
crews practiced CRM during normal line flights. With my colleague John Wilhelm, 
retired Pan American World Airways captain Roy Butler, and a team of trained 
observers, we collected data on crew behavior during regularly scheduled flights. To 
code observations we adapted the systematic observational methodology that I had 
employed studying the behavior and performance of aquanauts living in a habitat on the 
ocean floor in Project Sealab (Radloff & Helmreich, 1968) and that John Wilhelm and I 
had used in observing the behavior of aquanauts living on the bottom of the Caribbean 
in Project Tektite (Helmreich, 1972, 1973). We observed 291 Delta domestic and 
international flights. The results were most reassuring: Delta crews were practicing 
CRM on normal flights as evidenced by their effective use of the behavioral indicators of 
good CRM. 

The observational methodology we employed evolved into the Line Operations Safety 
Audit (LOSA) under the guidance of James Klinect, PhD, a graduate of our program 
and principal of the LOSA Collaborative. CRM is an essential component of LOSA. 
LOSA’s strength is in the use of expert observers riding the cockpit jumpseat with total 
assurance of confidentiality to capture not only real time behaviors including task 
performance and CRM practices of crews but also the context of behavior and the 
outcomes—errors committed or managed and threats managed or mismanaged. LOSA 
and CRM have been mandated by the International Civil Aviation Organization for 
all the world’s airlines ICAO, 1998, 2002). 

LOSA in the USA was nearly sabotaged by the terrorist attacks on the World Trade 
Center in 2001 following which an FAA edict specified that only crewmembers 
could have access to the cockpit during flight. Continental Airlines responded to this 
situation by giving me an ID showing me in full captain’s uniform, although they were 
wise enough not to let me fly one of their aircraft. 

CRM rapidly infiltrated other components of the aviation system—soon we had 
Dispatch Resource Management and Maintenance Resource Management addressing 
team and inter-group issues. CRM training for air traffic controllers also emerged. 

After Southwest Airlines had completed initial CRM training for its pilots, I 
presented the results (observations and attitude change) to management. Southwest 
CEO Herb Kelleher attended and rose to speak after presentations by me and the 
managers and instructors of the CRM program. Herb said that it was not fair for pilots to 
be the only beneficiaries of such training—thus was born Management Resource 
Management at Southwest Airlines. 
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1.11.2. Acquiring and Using Safety Data 


Any successful program designed to improve CRM attitudes and behaviors needs to 
be based on valid data. As we have noted, the CMAQ and later the FMAQ provide 
reliable baseline information on the cognitive acceptance of CRM. LOSA, with 
guarantees of anonymity for those observed, provides a real-time snapshot of actual 
behavior. Another source of data also yields unique insights into organizational practices 
and CRM—confidential incident reporting systems. The Aviation Safety Reporting 
System (ASRS) managed by NASA has been in existence for more than 30 years and 
has amassed an enormous national database of events, but ASRS reports lack 
organizational specificity and don’t give airlines useful information on conditions in 
their own organization. American Airlines, under the leadership of Dr Thomas 
Chidester, then at American and now at the FAA, helped institute a local reporting 
system, the Aviation Safety Action Program (ASAP: AC 120-66, Federal Aviation 
Administration, 2002), which provides protection from disciplinary action for those 
reporting threats to safety and errors to their own organization. These reports are 
processed at the organizational level and provide useful insights into local issues. An 
ASAP committee including management and pilots’ association members reviews each 
report and develops a strategy to deal with the issues raised. A high percentage of ASAP 
narratives deals with CRM issues. Data from these sources combined with data-driven 
CRM training contribute to the development of an organization’s safety management 
system and safety culture (Helmreich & Merritt, 2000). 


1.11.3. Expansion of CRM into New Domains 
Medicine 


In 1994 I met an anesthesiologist, Hans-Gerhard Schaefer, from the University of Basel/ 
Kantonsspital in Switzerland. Hans had heard of CRM and decided that it might be just 
the thing to improve teamwork in the operating theaters of Basel. Hans traveled to 
Austin and spent a year in our lab at the University of Texas. During his stay in Texas he 
observed all aspects of teamwork and team training in aviation. Following his return to 
Switzerland, I was invited to spend a year as a visiting professor in Basel where, assisted 
by Bryan Sexton, a student of mine from the University of Texas, we observed physician 
and staff behavior in operating theaters during surgeries. We also participated in 
development of a Critical Incident Reporting System (CIRS) to allow professionals to 
share information on safety-related issues—especially CRM issues surrounding the 
interfaces between surgeons, anesthesiologists and nurses. 
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A few years later the United States Institute of Medicine (IOM) issued a highly 
influential report documenting the scope of preventable medical error. The IOM report 
concluded that more than 90,000 people a year may die needlessly in the USA from 
preventable medical error (Institute of Medicine, 1999). Comparing medicine and 
aviation, I discovered many similarities between the two professions. Stunned by the 
implications of the data, a number of medical organizations began to realize that they 
might benefit from adopting aviation’s approaches to safety (Helmreich, 1997). The 
British Medical Journal, one of the most prestigious medical publications, placed a crashed 
aircraft on the cover of its issue containing articles by me and others about adapting 
aviation safety approaches to healthcare (Helmreich, 2000). Contrasting death rates from 
errors in the two professions, it is apparent that your doctor is more likely to kill you than 
your pilot. The data also suggest that significant improvement may come from 
embracing aviation’s safety strategies including CRM (Helmreich & Sexton, 2004a, 
2004b; Thomas & Helmreich, 2002a, 2002b). 

Facing the reality of becoming an increasing consumer (and potential victim) of 
the healthcare system as I age, I became more involved in patient safety issues and in 
designing appropriate CRM training for healthcare professionals. In the USA, one of 
the barriers to the effective information exchange needed to optimize CRM in 
medicine is that, unlike aviation, there is no immunity from punishment or 
malpractice lawsuits for those who report and acknowledge their errors. Indeed, in 
Texas until recently a nurse who committed an error, even the administration of the 
wrong medication because of an error in the pharmacy, faced potential loss of 
license. The workaround for lack of protection for those who disclose errors has been 
to limit reports submitted to threat and error databases to near misses with no adverse 
impact on patients. I do not see this as a critical problem because near miss data 
usually have as much diagnostic value as information from events with less happy 
outcomes. In the absence of a more coherent healthcare system, it remains to be seen 
how useful these data will prove to be and if medical CRM training enhances safety 
significantly. 


Firefighting 


Of all the professions in the USA firefighting has the second highest incidence of line 
of duty death (behind mining) with 114 fatalities in 2008. CRM training has been 
provided for firefighters to help them cope as individuals and teams with complex, 
dangerous and frequently changing situations where information is often incomplete. 
I had the privilege of working with the International Association of Fire Chiefs as they 
developed and implemented a national, internet-based close-call reporting system 
(www.firefighternearmiss.com). Their firefighter reporting system asks respondents to 
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identify multiple causal and contributing factors and to provide a narrative describing 
the event. Contributing and causal factors in the reports provide insights into team 
coordination issues and decision making. In larger fires there are frequently multiple 
units from different stations on the scene. This type of situation requires effective 
leadership as well as inter- and intra-team coordination. 


1.11.4. The Future 


I have been amazed and delighted at the proliferation of CRM in extremely diverse 
professions. The basic concepts of CRM clearly address critical safety issues. Cooke and 
Durso (2007), in their assessment of failures and successes, apply psychology to settings as 
different as minefields, the operating room, and the performance of elderly drivers. I feel 
confident that, in its threat and error management identity, CRM will continue to play 
a significant role in the training of professionals who work in areas where teams must 
interact successfully for safe and efficient task performance. 
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Introduction 


I have chosen to write this chapter in the first person. I prefer this style of writing 
because, unlike so many of the other authors of this book, I am neither a scientist nor an 
academic. Most of my opinions and observations are clinical, a result of long experience 
in the aviation industry. Both as a pilot and flight operations manager, I have held many 
positions within a major airline and served on numerous industry councils, committees 
and task forces. What I hope to add to this discussion is my perspective on the evolution 
of human factors analysis in the commercial airline business over a period of four 
decades. 

First, I would like to discuss the evolution of teamwork, most commonly called 
Crew Resource Management (CRM), and the many meanings and misconceptions the 
term has acquired during the past decades. A new definition of CRM removes all the 
ambiguities and clearly identifies the meaning and focus of CRM. This discussion leads 
to the recognition and definition of good leader attributes and to the realization of the 
ubiquity of errors in the aviation industry. There will also be a discussion of the 
importance of Standard Operating Procedures (SOPs) and the role they play in the safety 
culture of organizations. 

Second, I will address the organization and clearly define the different cultures that 
ultimately make up a true safety culture. Once again I must refer to leadership, since 
a safety culture can only start at the top of an organization and will not permeate the 
entire organization unless it has the support and backing, in writing, of the leaders. 


2.1. Updating the Definition of CRM 


When I first left the Air Force to fly for Continental Airlines, I noticed little difference in 
the way crew performance was evaluated. Commercial pilots were judged using the 
same tried-and-true measures of military training: how well does the crewmember 
handle the aircraft, know the rules and performance data, and deal with contingencies. 
The primary focus on training was the elimination of errors, and the primary tool was 
the check ride, in which the crewmember passed or failed based on his or her individual 
performance. The check pilot’s (and his flight operations superiors’) main task was to 
“wash out” those individuals who did not have the “right stuff” (to use the term made 
famous by author Tom Wolfe (1979)). The bottom line of training was to eliminate, as 
much as was humanly possible, all “human error” from the cockpit. 

Now, over 40 years later, the industry presently evaluates a cockpit team’s 
performance or teamwork using “Crew Resource Management” (CRM). Summed up, 
this new standard can best be explained in a statement made by a former FAA 
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administrator who said, “pilots do not cause accidents—crews do.” The unspoken 
concept behind his statement is that line managers need to look beyond an individual’s 
performance and judge that person’s skills as a team member. 

Granted, we cannot completely disregard individual performance. However, this 
matter is best addressed when an individual is first hired and/or during training for a new 
position or new equipment. All too much emphasis is still placed on this aspect of 
aviation training. In the new CRM model, individual performance should not be the 
focus during line checks, proficiency check/recurrent training, Line Operational 
Evaluation (LOE) or Line Oriented Flight Training (LOFT), rather the focus should be 
on how well the individual works in a team. Once an individual has been successfully 
qualified in a crew position the emphasis should then become how well he or she can 
predict or prevent errors through threat analysis, detect the inevitable errors made by the 
crew and correct those errors before negative consequences occur. The true definition of 
“teamwork” or CRM is its focus on the proper response to threats to safety and the proper 
management of crew error. 

The focus on “threat and error management” (TEM) does not mean a lowering of 
performance standards. While we accept the inevitability of errors we must nevertheless 
maintain performance standards. Error management demands that we distinguish 
between an individual’s recklessness or disregard for standard operating procedures and 
mistakes that are simply the product of human limitations. CRM requires that we reach 
beyond evaluation of individuals to that of the entire team responsible for safety in flight. 


2.2. Teamwork Redefined 


For our present purposes we need to make clear what constitutes a “team” in CRM. 
Broadly speaking, everyone who participates in moving a flight from A to B is a member 
of the team—including management, ground services, and even ATC. But for this 
discussion, the key members of a team are on board the aircraft, those who manipulate 
the controls and manage the aircraft systems, and handle the human and other cargo 
from gate to gate. In other words, the team consists of the cockpit crew and flight 
attendants. 

There is in addition another member of the team onboard the flight: the fitted 
equipment designed to reduce workload and increase safety in coordination with the 
flight crew. This “member,” usually described using the simplistic and misleading term 
“automation,” and thought of as “dutiful and dumb,” has become increasingly important 
in CRM due to the reduction in crew size and the increasing complexity of automated 
systems onboard. In present day aircraft this silent member of the team that accomplishes 
so much of the work in the cockpit will dutifully do any task it has been asked to do, 
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whether it makes sense or not. Failure to understand and integrate this member into the 
flight team has, since the introduction of automation, provided many painful lessons of 
the need to include this cockpit resource into flight management. 

From the beginning, the focus of CRM has been the attitude, behavior and 
performance of individual pilots. The objective was to eliminate the “wrong stuff” pilot. 
Good CRM was defined as a captain that creates an atmosphere where crewmembers 
feel comfortable to speak up and state opinions, ask questions and challenge if necessary. 
Indeed, the captain should insist on this behavior and praise it when it is present, not 
only talk the talk but also walk the talk. As early in the flight as possible, the captain 
should look for a situation where the other crewmember(s) input information and use 
that instance to praise the crewmember, thanking him or her for the teamwork. This has 
a positive teambuilding effect and is most important in a crew that has never flown with 
each other before; but it is necessary even with a familiar crew. In this model the captain 
is charged with reinforcing good performance and helping other crewmembers improve 
their responses to threats and their recognition of errors. 

The other crewmembers are charged with speaking up regardless of the atmosphere 
created by the captain. This may, at times, require an aggressive posture by the crew- 
members, which flies in the face of the common misconception that good CRM is 
“getting along in the cockpit.’ Good CRM is recognizing and identifying threats, 
preventing errors if possible, catching those that will inevitably take place and to the 
extent possible, through resistance and resolution, mitigating the consequences of those 
that have occurred. 

The threat and error model shown in Figure 2.1 illustrates the flow of this process. 
The only terms that require some explanation are “RESIST,” which represents aviation 


Figure 2.1 Threat and error management model 
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safety systems in the cockpit, in the ATC controller’s suite and on the ground, which 
create resistance to errors. RESIST includes systems such as Enhanced Ground Prox- 
imity Warning Systems (EGWPSs), windshear warning systems on the aircraft and the 
ground, and a Traffic Collision Avoidance System (TCAS). There is also a Minimum 
Safe Altitude Warning System (MSAWS) in the control towers warning of dangerously 
low altitudes, just to mention a few. As a result of the success of these systems Controlled 
Flight Into Terrain (CFIT) is no longer the number one cause of fatalities in our 
industry. Increasingly important in modern flight management are the automated 
systems designed to “resist” threats (altitude deviation, conflicting traffic, menacing 
weather) before they become unmanageable. 

“RESOLVE” on the other hand is what the human brings to aviation safety— 
proficiency, experience, effective monitoring and communicating, etc. Together resis- 
tance and resolution filter out errors that may inevitably occur and prevent negative 
consequences. 

In the model illustrated in the figure, the top level “STRATEGY” may be thought 
of as “managing our future” by recognizing threats and creating error-blocking strat- 
egies in advance. If, STRATEGY notwithstanding, a threat goes unrecognized or an 
error occurs, the subsequent levels (RESIST and RESOLVE) may be thought of as 
“managing our past,” whether by catching errors or by mitigating negative conse- 
quences. These levels of error management may appear inferior, but in fact are at least as 
effective in generating good outcomes as the STRATEGY level. 

The industry has come to understand and accept the ubiquity of error in our 
complex and dynamic aviation system and I would like to think we are now moving 
toward a robust error management system where a well-trained and focused crew can be 
very effective in accomplishing safe, economical and efficient flight. 

In the early decades of aviation the model of cockpit “pecking order” had it that the 
captain, like the 19th century ship captain, acted like a monarch in charge of his small 
kingdom. The co-pilot was the pilot not flying (PNF), sometimes given smaller tasks 
when the captain doled them out—handling the landing gear, flaps and communication, 
etc. In the 21st century the industry, for the most part, has now come to prefer the term 
pilot monitoring (PM) versus pilot not flying (PNF) to indicate the crewmember not 
primarily manipulating the controls. The implication of this change in terminology is 
that the PM is, in fact, an active participant in crew operations and is eminently as 
responsible for the safe conduct of the flight as is the pilot flying (PF). 

It is time we change the emphasis of the team and refocus on their resistance to 
error. Moving away from the individualistic accomplishment culture toward a true 
team accomplishment, culture is indeed a very hard task, for it is truly not only 
embedded in our national, industry and organizational culture but is also part of our 
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basic human makeup. This individualistic orientation begins at the very onset of 
a pilot’s career and is constantly reinforced in proficiency checks and even during 
advancements to new positions in the currently flown aircraft or moving into new 
aircraft. A pilot would much prefer to be known as a “good stick” rather than a good 
team member. This makes the formation of a good error detecting team all the more 
difficult to create. 

In addressing leadership skills and traits in this chapter I would like to share one of my 
life’s learnings: I can find no difference in the attributes or traits of a good leader or 
a good follower—they are the same. There are no absolutes in this world but my 
experience has been that a good follower will make a good leader and vice versa. I can 
think of many instances in my experience where the role of leader and follower flowed 
back and forth between crewmembers as the flight progressed. There is no argument 
that the captain is the team leader and will make the final decision but there will be times 
when he or she will be, and should be, in a follower role. That being said, I would like to 
outline some of the traits of a good team member, leader and follower. 


2.3. Traits of a Good Team 


Let’s take a look at the desired traits of a good team leader/follower. There can be little 
argument that the hallmark of an effective team is proficiency. The forgone conclusion is 
that each member of the team will be proficient at the task they are assigned to perform. 
This is the responsibility of the director of flight operations; the person responsible for 
ascertaining the ongoing proficiency and competency of the individual assigned 

a position in the team. This is something that airlines do very well. The training 
departments of the large airlines are extremely effective in turning out exquisitely 
prepared crewmembers and the certified training organizations, which do so much of 
the training for smaller outfits, do just as well. This is further assured by the Federal 
Aviation Administration (FAA) who set the minimum standards required for positions 
within the cockpit. 

Proficiency must include the commitment to comply with Standard Operating 
Procedures (SOPs). This may not seem a serious problem, but Boeing statistics kept 
since 1959 have shown that deviation from SOPs is a contributing cause in over one 
third of all hull loss and fatality accidents worldwide. This deviation may take the form of 
omission, failing to do something that should have been done (perform a checklist), or 
commission, accomplishing an action incorrectly or doing something that should not be 
done (e.g. checklist from memory, descent below minimums). The slippery slope here is 
that every time a deviation from SOPs is successful, it reinforces the act of getting away 
with it. This can lead to the “normalization of deviation” where the crewmember 
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doesn’t even recognize it as a deviation because it has been done so often and sometimes 
by so many. Deviation from SOPs is an ever-present problem in our industry and must 
not be tolerated by aviation organizations and agencies. 

Just as important as proficiency is effective communications. This, of course, includes 
the ability to communicate between team members and between the team and others 
outside their environment. Automation in our industry has increased the need for 
effective communication between members of a team. As we will discuss later, auto- 
mation does so much of the handling of the aircraft that it becomes extremely important 
that the crewmembers verbalize, verify and monitor any instructions or changes to 
instructions given to automation. 

Miscommunication between the pilot and the controller is the leading item cited 
in the NASA-managed Aviation Safety Reporting System (ASRS). Failure to 
communicate clearly can be especially dangerous and has been cited as a causal factor in 
a number of major accidents. Some of the worst examples of teamwork have been 
characterized by poor communications between the team members and also between 
the crew and those outside the cockpit. Conversely, one of the best examples of superior 
communication was the United DC-10 accident at Sioux City, lowa. The crew, 
including a company pilot who was riding in the back, did a remarkable job of 
communicating within and outside the cockpit and was able to bring the flight to the 
best possible outcome considering the horrible situation. Robert Helmreich of the 
University of Texas studied the cockpit voice recorder (CVR) and judged that at some 
points in the emergency the crewmembers were processing as much as one item of 
information per second, a remarkable accomplishment considering the amount of stress 
under which the crewmembers were working. 

Effective communication can take many forms, both verbal and non-verbal. One 
of the most effective verbal communications takes the shape of briefings and debriefings 
of the entire crew. As the complexity of the industry has grown, preflight briefings 
have become increasingly more important, both between the company (dispatcher) and 
the crewmembers and also within the team itself in the form of a preflight briefing 
when the entire team comes together prior to the flight. Given the size of some of the 
crews in modern aircraft this can be a daunting task but is all the more important as 
the crew size increases. These briefings are necessary to clarify the task responsibilities of 
the crew and the environment in which the flight will be conducted. This will assure the 
entire crew is of the same mindset. Debriefings are also a very important part of 
communications. It is the best opportunity to highlight and praise good teamwork and 
also point out areas needing improvement. Remember, the most important thing 
when debriefing a negative event is to emphasize “what went wrong” not “who was 
wrong” and how do we prevent it from happening it again. 
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Effective monitoring is also one of the more desirable traits of an effective team. 

As mentioned earlier, the industry has embraced the concept of the pilot monitoring 
(PM) versus the pilot not flying (PNF). This is a subtle change but the implications 
are large. The PM has many tasks to accomplish in support of the PF but the primary 
job of the pilot monitoring is to monitor the progress of the flight and PF’s performance 
to detect any threat or error that can lead to negative consequences. If a threat or an 
error is detected, that crewmember’s job then becomes an assertive challenge that will 
identify the threat so the error does not occur, or identify the error so there are no 
negative consequences. 

A study identified the failure to monitor and challenge by low-time-in-type 
co-pilots as being especially prevalent (Flight Safety Foundation, 1994). Whether this 
was the result of an inexperienced pilot who didn’t monitor the error, or the insecure 
pilot who saw the error but failed to challenge, will never be known. The study was 
backed up recently with similar results (Dismukes et al., 2007). Both of these studies 
highlight the need for training and evaluating monitoring skills. 

Monitoring is a skill that has to be trained, practiced and evaluated. This is 
something that has not been a large part of training programs to date, yet is so important 
to the successful accomplishment of threat and error management. One of the most 
effective ways of providing pilots with a motivation to be good monitors is to evaluate 
and hold them responsible for that skill in a training setting. Line Oriented Flight 
Training (LOFT) and Line Oriented Evaluation (LOE) sessions are the ideal vehicles for 
accomplishing this. However, monitoring should be evaluated anytime a crew is being 
observed. The need for good monitoring skills can easily be emphasized during any 
training session. When an error is made by the PE the instructor should ask the PM why 
he or she allowed the error to occur, a major shift in the way crewmembers are trained. 
This accomplishes two things. First, it identifies the primary role of the PM but also 
de-emphasizes the focus on the individual performance and refocuses it on the team and 
the role of each team member. 

Included in the skill of monitoring is vigilance and, more important, knowing when 
to be vigilant. A crew cannot and need not be vigilant all of the time. Knowing when to 
be vigilant and when to relax a little is important especially in today’s environment with 
aircraft having such long flight endurance including Ultra Long Flights (ULFs). 

Another important trait of a good team is modeling. It is an excellent way for a leader 
or follower to demonstrate a personal example of compliance with all standard operating 
procedures. Modeling is a method of giving positive feedback or sharing knowledge 
without appearing to critique or give a “flying lesson.” It was Albert Einstein who said 
“Setting an example is not the main means of influencing others; it is the only means.” 
Conversely, there are few things worse in this industry than presenting a bad example. 
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A role model or leader does immeasurable damage when he or she does not adhere to 
standard operating procedures. The “do as I say not as I do” performance by a person in 
a position of power can have a negative effect far in excess of the one incident that is 
observed and can add to the previously mentioned dangerous problem in air trans- 
portation, the “normalization of deviation,’ The damage that can be done by even one 
individual with this attitude is considerable. The modeling of dignity and the respect of 
conduct consistent with standards is a very powerful tool that can and should be used by 
a leader and a follower. 

Envisioning is just another way of saying what pilots have heard from their first flight: 
“stay ahead of the aircraft.” This skill creates and shares a plan for the entire crew and is 
an absolute necessity for good situation awareness (SA). It supplies meaning and 
direction for the task at hand and along with a good briefing creates operational clarity 
and sets workload management parameters. One of the more meaningful sayings heard 
in aviation circles is “never take your aircraft any place your mind hasn’t been five 
minutes earlier.’ This in a nutshell is envisioning. 

Leaders and followers must also be adaptable. The ability to adjust to changes is an 
absolute necessity in air transport operations. “Decision bias and plan continuation” has 
been cited as causal factors in many aircraft accidents over the years. The resistance to 
change is something inherent in all humans. The adjustment to changes that occur in 
flight and the willingness to build and share a new plan are hallmarks of good airmanship. 
Teamwork requires a balance between structure, which all humans require, and the 
ability to recognize a changing environment requiring the flexibility to adjust as 
necessary. 

When a crewmember pays attention to others’ ideas, concerns and or questions, 
that person is demonstrating receptiveness. This skill along with adaptability and the 
willingness to change are key elements of a safe cockpit. Listening to suggestions and 
adopting the suggestion when appropriate is a very powerful teambuilding tool. Even 
when one disagrees with the idea it is an important part of crewmember team- 
building. The recognition of the input and the appreciation that should be voiced by 
the receiver of the input will strengthen the cohesiveness of the crew. There will be 
times when a suggestion or concern is not adapted during flight but it is important 
that it be acknowledged even if the end result is the “agreement is to disagree.’ The 
cockpit is not a democracy. The captain is the final authority and will make the final 
decision. 

Using logic and tact, a leader/follower can influence others and obtain a commitment 
to ideas or actions. This is an especially important attribute when faced with incomplete 
or conflicting information from diverse sources and there is a need to assess novel 
situations and devise appropriate solutions. Situations like this require the use of all 
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available resources and may include a junior crewmember being assertive enough to 
convince a senior crewmember into a desired action. 

And finally, when a crewmember begins an appropriate action, within bounds and 
without direction, that person is exhibiting initiative. This leader/follower attribute is 
especially important when an action is begun to correct an operational deficiency. This 
may take the form of a pilot tactfully correcting another pilot when that person is doing 
something non-standard; or it may be an action taken to find a more effective way to 
accomplish something, again within bounds. 


2.4. Errors 


Errors are an inevitable part of flying. We in the aviation industry have been laboring 
long and hard to eliminate as many errors as possible and have done a reasonably good 
job. Truth is, however, that error is ubiquitous and probabilistic and can never be 
eliminated completely. New systems (resistance in the TEM model) installed in our 
aircraft and in ATC suites have gone a long way toward protecting the pilots and 
controllers from error-producing situations. This is borne out by the remarkable safety 
record that our industry has been enjoying over the past decade. However, the fact 
remains that we will never be able to eliminate all errors in an industry as dynamic as air 
transportation. 

We live in a blame society. This is most evident when a negative event occurs. The 
hunt for the individual villain begins immediately. Conversely, the positive event results 
in a similar hunt for the hero. Too often we find the attitude that the crew caused the 
accident because they made errors central to the events leading up to the accident. It 
appears that we are convinced that human error is a cause of trouble in an otherwise safe 
system, when in fact I’m convinced that human error is not a cause but a symptom. It is 
a by-product of hard working crewmembers trying to pursue success in a resource- 
constrained, uncertain, imperfect system. That is the underlying assumption in error 
management. Any time a human is used to operate equipment, no matter how well 
selected, how well trained, or how optimally used, the human is subject to limitations. 
The flip side of human performance is human error. 

Jerome Lederer, a leading pioneer in aviation safety, said the following in a lecture 
given to the Royal Aeronautical Society in 1952: 


... [he average man has only one head, two eyes, two hands, two feet, his response to 
demands cannot be guaranteed within plus or minus five per cent; his temperature cannot be 
allowed to vary more than a few degrees; his pump must operate at constant speed and 
pressure; his pressure containers, both hydraulic and pneumatic, have limited capacity; his 
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controls are subject to fatigue, illness, carelessness, anger, inattention, glee, complacency and 
impatience. This mechanism was originally designed to operate in the Stone Age; it has not 
since been improved. The problem consists of permitting this ancient mechanism designed to 
function within narrow tolerances to control its destiny in a strange environment of very wide 
ranges in operating conditions. 


Recognizing this limitation, a change has to occur in the way we train and evaluate 
our crewmembers. The performance of a team should not be based on error-free 
operation but instead the emphasis should be on threat recognition, detecting errors and 
managing, to the extent possible, the consequences of errors. Traditionally, flight 
instructors have trained by rewarding error-free performance. Errors have always 
counted against a pilot. The consequences are usually lower grades, further training, 
debriefing, etc. This is often true even if an error is caught before it becomes a serious 
problem. In the conventional approach to training and evaluation the fact that an error 
was committed becomes the center of attention. 





More is known about a crew that makes an error and manages it than is known 
about the crew that doesn’t make the error. 


At the heart of this is a desire to eliminate, as much as possible, errors during 
instruction and evaluation. If there were a finite number of errors that could occur in 
aviation it might be possible to train crewmembers to eliminate them. However, history 
has shown us that the number is infinite. Murphy’s law is alive and well in our industry 
and if something can possibly happen, it will happen. So the challenge is to create an 
error management system in which the crewmembers recognize threats that can cause 
errors, guard against the errors that will inevitably occur and correct errors before there 
are any negative consequences. More is known about a crew that makes an error and 
manages it than is known about the crew that doesn’t make the error. 

The way that instructors conduct training and evaluating is extremely important. 
Instructors have to move away from the “blame and train” method of training and 
concentrate on the crew’s ability to recognize threats, detect and manage errors. The 
instructor still must detect errors made by the crew, and eventually point them out. 
However, when crewmembers detect and resolve an error quickly, for all practical 
purposes, the error did not occur. This is exactly the behavior we desire in our industry and 
it must be recognized and rewarded. In good error management training it is possible for 
a crew to make an error, detect and resolve it and actually be graded higher on that 
particular event or the entire training period. 
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As I stated earlier, effective monitoring by the crewmembers is an all-important 
element of error management. In training, the pilot monitoring (PM) should be held 
responsible for monitoring the performance of the pilot flying (PF) and maintaining 
situation awareness of the progress of the flight. This implies that there will be a grading 
system established whereby monitoring can be graded. 


2.5. Standard Operating Procedures (SOPs) 


Airline management must provide great clarity about the day-to-day task responsibil- 
ities of crewmembers; these tasks and how they are to be performed must be spelled out 
explicitly in great detail in flight operations policy manuals. The importance of good, 
well-thought-out SOPs cannot be over emphasized. My experience has shown that the 
degree to which these procedures are adhered to is usually a good measure of the 
quality of the airline and a very good measure of the quality of the airline’s instructors, 
check airmen and management pilots. It is also an indication of the quality of the 
captains of the airline. If junior officers are found to be deviating from SOPs it’s a strong 
indication that the captains with whom they have been flying have allowed these 
deviations and not taken action to stop this type of behavior; or worse, they are 
deviating themselves. I firmly believe that a behavior uncorrected is a behavior 
condoned. 

If crewmembers are not given clear direction on how tasks are to be performed they 
tend to “do their own thing,” and this can have dire consequences. Even when the task is 
clearly defined there are crewmembers that will deviate from normal procedures. 
Psychologists learned long ago that the more ambiguity there is in a situation, the more 
personality differences show themselves (Dismukes et al., 2007). 

Standard operating procedures ensure that crewmembers that have never flown 
with each other before will come together with the knowledge that the flight will be 
flown by the book, and know exactly what to expect from each other. The procedures 
set forth in flight operations should be time-tested methods of assuring the flight will 
be safely and efficiently flown. This can only be accomplished if the procedures make 
sense to the pilots and they feel they have a stake in the formation of the procedures. 
To get buy-in from the crews, it is extremely important that these procedures be 
reviewed on a regular basis and that there is line pilot input. The feedback stream 
should be from the line pilot upward through management and every recommen- 
dation should be considered carefully. At the same time it is incumbent on the 
management of the organization to check and evaluate all crewmembers on their 
adherence to SOPs. Deviation from SOPs cannot be tolerated and should be dealt 
with on the spot and, in some cases, disciplinary steps may be necessary. 
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Of course, there will be times when the crew has to use whatever means they deem 
necessary to accomplish the flight safely. This is addressed in the Emergency Powers of 
the Captain granted by the FAA in federal regulations. There will also be times when the 
crew has to use their leadership skills to deal with an ambiguous or unique situation. 
Management can assist the crewmembers by providing clear and challenging direction 
about the desired end-states that ensure efficient, on-time performance and customer 
satisfaction. This is when the ability of the human far surpasses that of a computer. The 
creative and innovative abilities of the human to deal with this type of dilemma and make 
split second decisions in situations never faced before is when a crew “earns its pay.” One 
of the most remarkable and recent examples of this is the “miracle on the Hudson” 
when the crew of a US Airways Airbus 320 made a successful, rarely ever before 
accomplished, water ditching of a modern jet aircraft. The successful accomplishment of 
this extremely unusual emergency water landing with no loss of life is an astonishing feat 
and speaks to the professionalism of our pilots and flight attendants. 


2.6. Organizational Factors 


I believe we can all agree that each organization has its own individual culture; however, 
my experience has been that there can be many subcultures within an organization. It is 
expected that an organization would take on the personality of its leader, good or bad. In 
general, I have found this to be true but there are many mid-level managers that 
certainly create subcultures within the major organization, again good or bad. How well 
an organization recognizes and deals with these subcultures is a measure of the health of 
the organization and its safety culture. I will discuss and clearly define a safety culture 
later in this chapter. But first I would like to discuss the development and refinement of 
Standard Operating Procedures (SOPs), which is the first step in creating a safety 
culture. 


2.7. Developing SOPs 


The flight department of any organization has to create a crystal clear set of guidelines 
governing flight operations. This is separate from the organization’s “vision” and 
“mission” statements, although these documents will still play a role in the operation of 
the flight department. This can be accomplished in four steps as shown in Figure 2.2, 
which follows a well-developed human factors paradigm known as the “Four Ps” 
(Degani & Wiener, 1994). The first step is for the most senior management person in 
the flight operations department to create a philosophy; this is usually the vice president of 
flight operations in a major airline. This should be a broad statement outlining, in 
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Figure 2.2 Four Ps model 
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general terms, how the department should go about conducting safe, efficient operations 
which ensure regulatory compliance and customer satisfaction. This can also include 
wide-ranging guidance on the use of automation. In fact, it was the introduction of 
highly automated aircraft that created the need for a philosophy of automation, which 
defined the different levels of automation available and their use under varying 
circumstances. This is certainly a necessary piece of the overall philosophy. 

The second step is for the next level of management, usually directors, to draw upon 
the broad philosophical statement to create policies that further define the desired goal 
and focus on the methods used to accomplish the desired outcome. It is important to 
note that this level of management is closer to the everyday operation of the organization 
and therefore the appropriate level to create policy. Upper management should delegate 
this task to the directors and not micro-manage the accomplishment of the policies. 
Micro-management is the quickest and surest way to create problems in a flight 
department. 

The third step is for the managers, supervisors and flight instructors to create 
procedures that complement the policies set forth by the directors. It is of utmost 
importance that the crewmembers on the line play a role in creating these procedures. In 
addition, the procedures must be constantly reviewed and revised if necessary. The 
model should end at this point but real world experience shows there is often 
a disconnect between procedures and the next and fourth step—practices. This discon- 
nect is also known as procedural non-compliance and can be prevalent in any organi- 
zation that doesn’t work hard to minimize it. Practices are the measure to which the 
procedures have been accepted and are followed by the average crewmember. Another 
definition of practices is “how it’s really done around here” or “norms.” Norms are 
described as “a practice bought into or tolerated by the majority” and are very hard to 
reverse. They may be so prevalent that crewmembers are unaware that they are devi- 
ating, and they may be around for years, undetected, until something happens that 
highlights their existence. James Reason labels this a “latent failure” or hazard (Reason, 
1994). Once a norm exists it is very hard to rid the organization of it. 
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You will notice in Figure 2.2 that the arrow between procedures and practices goes 
both ways. This is an important flow of information that serves to minimize procedural 
deviations. Managers and supervisors should constantly watch for procedures that are 
not being followed and look for the reasons. It may be that crewmembers have devised 
a better way to accomplish the task and it should be adapted as a procedure; this is a win— 
win situation. Of course, it may also be that crewmembers have found an easier or 
quicker, but not necessarily a safer, way to get the task done. Procedural deviation is hard 
to detect and rarely surfaces during normal checks such as proficiency checks or yearly 
line checks because crewmembers are at their best behavior during these occasions. One 
of the most effective ways of detecting them is during audits such as Line Operations 
Safety Audits (LOSAs), a non-jeopardy audit that many airlines around the world have 
adopted for just this reason. I will speak of LOSAs when I next describe a safety culture. 


2.8. Safety Cultures and Organizations 


There is one extremely important rule about an organization’s culture—it is created at 
the top and permeates the entire organization. However, it is always measured at the 
bottom where the work is being done. This fact points out a critical aspect of safety 
cultures—the highest level of management must be fully committed, lead the way and be 
the loudest and strongest proponents of the safety culture and all its ingredients. A recent 
article in Aviation Week and Space Technology stated the following: 


Investing the time and money needed to get at the root cause of a problem takes total 
commitment at the most senior levels of a company or organization. In most 
organizational settings, communicators learn early in life how bad news can impact their 
leaders. If the news is valued and the communicator is protected, there is a real 
chance information can and will routinely flow upward in time for proper action 
to be taken. 


The unspoken word here is trust! Company personnel must feel protected and this 
protection should come, in writing, from the highest levels of the organization. 
Non-jeopardy programs are the finest examples of this type of protection. This does 
not mean that willful violations or dangerous and reckless behavior will be tolerated; this 
type of behavior should be acted upon harshly and swiftly. These programs do recognize, 
however, that learning how, where and when a human error has occurred is much 
more important than placing blame, and can be useful in preventing a reoccurrence. 

As I have stated, we live in a blame society. This means that whenever there is 
a negative event there is always a rush to find the villain or villains. This need to place 
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blame gets in the way of the ultimate objective of preventing the event from happening 
again. Corporate leadership and middle managers must recognize that the value of this 
information far outweighs the small satisfaction we get from placing blame and 
punishing. 

One of the most effective programs of this type in current use at many major air 
carriers is the FAA Aviation Safety Action Program (ASAP), a non-jeopardy part- 
nership that encourages corporate-specific voluntary safety reporting. Management 
and unions play a vital role in these programs and their support cannot be over- 
emphasized. I have always believed that a strong union is an asset for an organization 
with the skill to use this vital resource and the wisdom to capitalize on their talent. 
There is no stronger program in an organization than one that has the combined 
backing of management and union officials. Unions usually have committees that deal 
with safety, training and professional standards. These groups can play a crucial role in 
the effectiveness of the operation; management should make the most of this source of 
manpower and intelligence. 

Another excellent source of support is the Air Transport Association (ATA) and for 
international carriers, the International Air Transport Association (IATA). These 
organizations and their councils, committees and task forces are invaluable resources 
available to an air carrier. They have led the way in many of the safety initiatives of the 
past few decades, such as windshear and traffic collision avoidance systems, and have 
been successful in lobbying in favor of beneficial programs for our industry at the 
government level. 

The subject of corporate culture has been a much-discussed item for many years and 
our industry has seen startling examples of both good and bad. At a Symposium on 
Corporate Culture and Transportation Safety in 1997 the honorable Jim Hall, a former 
chairperson of the NTSB, had these comments on the subject: 


We have found through 30 years of accident investigation that sometimes the most common 
link is the attitude of corporate leadership toward safety. The safest carriers have more 
effectively committed themselves to controlling the risks that may arise from mechanical or 
organizational failures, environmental conditions and human error. 


2.9. Safety Culture 


A safety culture has been described as the product of the individual and group values, 
attitudes, competencies and patterns of behavior that determine the commitment 
to, and the style and proficiency of, an organization’s health and safety programs. 
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James Reason has taken this description one step further by breaking down the 
individual parts necessary to create a safety culture. Dr Reason says there are four parts 
to an effective safety culture, an informed culture, a reporting culture, a just culture 
and finally a learning culture (Reason, 1993). Let’s discuss each one of these 
individually. 

An informed culture is one in which an organization collects and analyzes the right 
kind of data to keep it informed of the safety health of the organization. This collection 
can be done in a number of ways. One of the easiest ways is to analyze data from the 
training department in the form of satisfactory and unsatisfactory performance during 
check rides. Depending on an organization’s grading system, there may be invaluable 
data from proficiency checks, recurrent training and LOFT performances that identify 
the need to develop focused training. For instance, items that are graded poor or 
unsatisfactory more often than other items in the check rides may indicate the need for 
emphasis on those particular items. 

Another excellent source of data for an organization is the FAA Flight 
Operations Quality Assurance (FOQA) program, which flags data from the digital 
flight recorder in flight that exceed certain parameters. If collected and analyzed 
correctly, the data will show trends such as flap speed exceedences, excessive speed 
below 10,000 feet, unstabilized approaches, etc. If a trend is detected, the organi- 
zation now has options on how to reverse the trend. This may not always be a flight 
crew problem, which can be addressed with bulletins and training. Experience has 
shown us that at times, a particular destination with a rash of flap speed exceedences 
may indicate a poor arrival profile. This is where union committees and ATA 
committees can be of great value. Sharing the data and working with other 
organizations experiencing the same problem in the industry have been very 
successful in resolving such problems. 

An organization with a robust informed culture can create a safety information 
system that collects, analyzes and disseminates information on incidents and near misses, 
as well as proactive safety checks. The key word here is disseminating. Information of 
this type is of no value unless it is sent through the proper channels and to the right 
people so that action is assured. 

A reporting culture is one where employees are encouraged to report safety problems. 
The most important ingredient of this culture is trust. They must feel confident they will 
not be punished or ridiculed for reporting. This trust can best be achieved if a written 
non-reprisal policy exists, signed by the most senior management as mentioned earlier in 
this discussion. This assumes confidentiality will be maintained or the data are de- 
identified. Lastly, they must have confidence the information will be acted upon if found 
to be meaningful. 
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A just culture exists if the employees realize they will be treated fairly. Recognizing the 
ubiquity of error, organizations will not punish those who error commit unsafe acts so 
long as the error was unintentional. However, it must be perfectly clear that those who 
act recklessly or take deliberate and unjustifiable risks will be punished. Willful violations 
and reckless operation will not be tolerated and will be acted upon swiftly and painfully if 
necessary. 

The final part of an effective safety culture is a learning culture. In short, the orga- 
nization is able to learn and change from its prior mistakes. This may seem an over- 
simplification but those who study management know how difficult change can be. 
Human beings are inherently resistant to change. The enemy of any organization is 
“business as usual.” Even after a problem has been identified and corrective action 
initiated, it is not unusual for the day-to-day operation to slip slowly back to the old 
routine. One of the definitions of insanity is doing the same thing over and over again 
and expecting a different outcome. Yet this is something we see organizations doing 
constantly. The ability to correct operations that are going wrong is truly a skill fraught 
with apprehension and angst. It takes a strong leader with a clear vision of what he or she 
wants and, even better, an understanding of how to get there and what it looks like to 
achieve true and lasting change. 

My experience has shown that although an organization can have an overall healthy 
safety culture it is possible for departments within the company to differ greatly. For 
instance, when a new aircraft type is introduced into an organization, the new fleet 
manager(s) usually incorporate the latest concepts in training for technical proficiency 
and human factors skills. This is a positive step forward because all aspects of the training 
and operation tend to be scrutinized and optimized. Using lessons learned by others 
already flying the aircraft type may also benefit the new fleet in creating the best possible 
procedures. 

However, the older fleet types don’t necessarily benefit from this optimization or 
may resist changing from the way they have historically done things. This is one of the 
easiest ways for cultures within an organization to drift apart. As more modern aircraft 
are brought into the fleet, the older aircraft fall further and further behind in the way 
they operate. This change is so slow and insidious that it is hard to detect. Sometimes this 
problem does not come to light until an incident or accident highlights the problem and 
the fix is a reactive one. 

One of the most successful ways of overcoming this problem is through a robust 
trend analysis and auditing system. Line Operational Safety Audit (LOSA) is 
undoubtedly one of the most effective ways to accomplish this task proactively. LOSA 
will be covered in depth by Bruce Tesmer in Chapter 10 so I won’t say much more 
except it is an excellent way to get a unique and insightful view of an operation. 
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2.10. Summary 


The theme of this chapter is that crewmembers and organizations can capably manage 
their errors. Error management is not a new concept; it has been around since the 
earliest days of CRM. In the early 1980s, I heard Clay Foushee (then of NASA Ames) 
use the term when talking about Line Oriented Flight Training (LOFT). Unfortunately, 
at the time, we in the industry were so focused on the “wrong stuff” pilot we didn’t 
appreciate the relevance of his statement. 

I hope this chapter will change the way we look at errors. We need to convince those 
in our industry who regulate, manage, train and evaluate crewmembers that accident- 
and incident-causing errors are actually symptoms of an imperfect system in which 
imperfect humans operate. If that can be established, we can maximize the effects of the 
newest version of CRM—threat and error management (TEM). This includes the 
concept that even the finest of crews can make errors, and when they occur they are able 
to trap and correct those errors, and should be subsequently rewarded for their actions. 

I have highlighted, if not exhaustively discussed, the optimum traits and attributes of 
crewmembers, the need for organizations to continually audit and evaluate their 
operation, and the many methods that now can be used to analyze trends in the industry 
and make systemic corrections. Hopefully, this chapter will provoke thought (and 
action) on how crewmembers are motivated, trained and evaluated. I look forward 
to the next generation of CRM and the many excellent ideas and concepts that are 
sure to come. 
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3.1. Individual Versus Crew Orientation 


On January 15, 2009, the “miracle on the Hudson” occurred. Shortly after takeoff from 
New York’s LaGuardia Airport, US Airways flight 1549 struck a flock of birds and the 
unthinkable occurred—both Airbus A320 engines lost power. In command, Captain 
Chesley “Sully” Sullenberger soon determined he would not be able to make 
Teterboro’s Runway 1 as assigned by New York’s TRACON and the best alternative 
would be to ditch in the Hudson River. The individual airmanship displayed by Captain 
Sullenberger in guiding what had become a commercial glider safely down to a water 
landing is unquestioned and truly remarkable. Individual airmanship will always be 
essential, especially in emergency situations. But even Captain Sullenberger has noted 
that the safety and survival of all 155 aboard was a crew accomplishment, and that notion 
is the essence of this chapter. 

The objective of this chapter is to change the focus of crewmembers from solely 
a perspective of competent individuals coming together to do work to a perspective that 
acknowledges that a crew, group, or team has certain unique characteristics that cannot 
be explained at the individual level. Further, these group concepts are critical for 
performance and should be understood and leveraged by anyone who considers leading 
a crew. 

To accomplish this objective, we will first look at some examples of crew failure and 
then introduce a few critical group-based concepts. Paramount among these will be 
group dynamics and leadership (yes—leadership is a group concept, not an individual 
concept). Then I will briefly review a NASA-funded research project I conducted 
examining the importance of leadership during the formation process of crews and 
discuss some of the unexpected results of that study. The concept of organizational shells 
will be introduced to help explain the surprising findings. Lastly, the implications for 
effective crew leadership will be discussed. 

A crew is a group and arguably the most critical resource in Crew Resource 
Management. It is also the primary and fundamental issue if we are to improve the work 
outcome for those who fly airplanes in the crew environment. But it goes far deeper 
than just the work in crew-served aircraft. Across the USA, we are discovering the 
difficulty of making the transition from individual work to group work in many of our 
industrial settings. 

Our tendency not to think in group concepts is itself a group issue. We are an 
individualistic culture (Triandis, 1995). From birth through college, we nurture and 
praise the individual accomplishments of our offspring. Whether in academics or 
athletics, in myth or in history, we focus on and reinforce individual accomplishment. 
This is not to say that group-oriented activity is ignored, but rather to say that we do not 
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focus as much attention on the accomplishments of groups as we do on the 
accomplishments of individuals. Being a member of the NCAA championship football 
team is obviously cause for celebration. But are we more inclined to remember the team 
that won the national championship five years ago or the winner of the Heisman trophy 
from five years ago? Being a member of the national collegiate debate team is something 
to be proud of—but in our culture being a Rhodes Scholar carries more prestige. 

Even our educational systems are based upon individual competition rather than 
group collaboration. At the US Air Force Academy a group of fellow faculty members 
and I came to believe that the entire system—from elementary school through 
undergraduate pilot training—evaluated and rewarded individual performance. At the 
same time, we began to recognize and acknowledge that once finished with the formal 
“training” portion of the lives of our pilots, the subsequent “work” which was to be 
done depended largely on the ability to work in a group. This notion was reinforced in 
the extreme when F-16 pilots from Nellis Air Force Base requested our research results 
on crew performance. As they noted, F-16 pilots work in “‘two-ships” or “‘four-ships,” 
and even though they were in separate cockpits, they needed to work as a group or team 
to be effective. 

We have imported and strengthened this individualistic orientation in the aviation 
community. From the early days of flight training, the goal is to “solo.” Iam hard pressed 
to come up with a more individualistic term than “solo.” Historically, the airline 
industry and those responsible for its oversight have been primarily interested in the 
qualifications and performance of the individual even though the individual was to be 
inserted into a crew-served cockpit. Airline companies have traditionally hired many of 
their pilots from the military, which assured them some reasonable minimum standard of 
training and experience in flying modern aircraft. Other pilots hired by the major 
companies have had to demonstrate comparable levels of qualifications. Likewise, the 
Federal Aviation Administration (FAA) certifies individual pilots on their technical skills 
at flying the airplane (for the captain and the first officer) or at managing the aircraft 
systems (for the flight engineer). For example, pilots are asked to demonstrate in 
recurrent simulator training procedures for difficult and infrequently encountered 
conditions, such as steep turns, multi-engine failures, recovery from windshear stalls on 
takeoff, and go-arounds in weather conditions with less than minimum visibility. 
Scheduling in most major airlines is driven by individual considerations, with seniority 
of the individuals in each of the positions being the principal factor. Only within the last 
few decades have we begun to consider this issue of crews and groups (which is quite 
foreign to our culture) in the training of teams that fly commercial aircraft. 

Before one gets the idea that this chapter is “anti-individual,’ let me lay those 
fears to rest. As noted in the opening paragraph briefly describing Captain 
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Sullenberger’s remarkable water landing skills in a circumstantial glider, nothing in 
this chapter suggests we need any less individual competence if we are to enhance 
crew performance. As we shall see later, individual skills are critical in aviation 
performance and should continue to be developed and rewarded. However, we have 
reached a point in aviation history (and in American business as well, I might argue) 
where we need to take the next step and go beyond the individualistic focus. That 
next step requires that we learn about groups. 

Sometimes we hear the argument that “groups are nothing more than the collection 
of individuals making up the groups.” Such statements ignore a growing body of 
evidence in both the research literature and in the annals of aviation mishaps. Rather 
than citing evidence from both of these sources, let me provide a very simple example to 
show how group work can be quite different (and to someone with little group 
experience, even counterintuitive) from individual work (Langfred, 2000). Again this 
example comes from athletics. 

As a culture built on valuing individual performance, we are sometimes given 
individual advice which will not necessarily result in quality team outcomes. For 
example, often team members are told by their coach that they all need to do their 
absolute best if the team is going to do well (at least, that is what my coaches told me on 
more than one occasion). But from systems theory we know that for a team to do well, 
sometimes the individuals comprising the team must not maximize their individual 
effort. Referred to as subsystem non-optimization, this concept is not intuitively 
obvious either to many team members or their coaches. But consider a high school 
football team which has an extremely fast running back and some very good, but 
measurably slower, blocking linemen. If our running back does his absolute best on 
a sweep around the end, he will run as fast as he can. By doing so, he will leave his 
blocking linemen behind. The team is not likely to gain much yardage on such a play, 
and the back, who has done his individual best, is apt to learn an important experiential 
lesson about teamwork. The coach would get better results if he or she worked out an 
integrated coordination plan between the back and the linemen. In this case, the fast 
running back needs to slow down (i.e. not perform maximally) to give the slower but 
excellent blockers a chance to do their work. After they have been given a chance to 
contribute to the play, the back will then have a much better chance to excel 
individually, and so will the team as a whole. Good teamwork is sometimes on a different 
plane (no pun intended) from good individual work. 

Unfortunately, we find repeated evidence of poor crew work resulting in 
errors, accidents and incidents in the aviation community. Three of the more 
publicized examples should be sufficient to illustrate this problem. The first 
example is taken from a National Transportation Safety Board investigation 
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(NTSB, 1979, pp. 23-29). It illustrates both the pervasiveness of the captain’s 
authority and the group’s failure to demand that attention be focused on a critical 
aspect of the flight: 


The crew of Flight 173 had experienced only routine conditions as they brought the four- 
engine DC-8 into the Portland, Oregon traffic pattern. However, on final approach as they 
lowered their gear for landing, they heard a dull thump from what seemed to be the main gear 
area. The captain elected to abort the landing and was put into a holding pattern until they 
could determine if there was a problem and whether or not it warranted further emergency 
precautions. 


The aircraft proceeded in a large holding pattern while the captain directed the crew in 
attempting to determine the possible cause of the noise. This pattern was maintained for 
approximately one hour at the captain’s insistence. During this time, both the first officer and 
the flight engineer warned the captain on four separate occasions that they were running out of 
fuel and needed to make a decision about landing. In spite of these repeated cautions, the 
captain insisted that they continue to circle. Finally, as the first of the four engines flamed out, 
the captain ordered the plane toward the field while demanding that the flight engineer 
explain the cause of the engine failure. With all fuel tanks now dry, the other engines began 
to fail in sequence and the DC-8 nosed downward. 


About 1815 PST, Flight 173 crashed into a wooded, populated area, killing 8 passengers 
and 2 crew members, and seriously injuring 21 passengers and 2 other crew members. The 
National Transportation Board determined that the probable cause of the accident was 

the failure of the captain to monitor properly the aircraft’s fuel state and to properly respond to 
the low fuel state and the crew members’ advisories regarding fuel state. This resulted in fuel 
exhaustion to all engines. Contributing to the accident was the failure of the other two flight 
crew members to fully comprehend the criticality of the fuel state or to successfully 
communicate their concern to the captain. 


The Safety Board believes that this accident exemplifies a recurring problen—a breakdown 
in cockpit management and teamwork during a situation involving malfunctions of aircraft 
systems in flight. To combat this problem, responsibilities must be divided among members of 
the flight crew while a malfunction is being resolved... 


Admittedly, the stature of a captain and his management style may exert subtle 
pressure on his crew to conform to his way of thinking. It may hinder interaction and 
adequate monitoring and force another crew member to yield his right to express an 
opinion. 


The second example, taken from a confidential report submitted to the NASA/FAA 
Aviation Safety Reporting System (ASRS) (Foushee, 1984, p. 888), describes a more 
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blatant example of an overbearing and intimidating captain. Here is the first officer’s 
report: 


I was the first officer on an airline flight into Chicago O’Hare. The captain was flying, we 
were on approach to 4R getting radar vectors and moving along at 250 knots. On our 
approach, Approach Control told us to slow to 180 knots. I acknowledged and waited for the 
captain to slow down. He did nothing, so I figured he didn’t hear the clearance. So I repeated, 
“Approach said slow to 180,” and his reply was something to the effect of, “I’ll do what I 
want.” I told him at least twice more and received the same kind of answer. Approach Control 
asked us why we had not slowed yet. I told them we were doing the best job we could and their 
reply was, “You almost hit another aircraft.” They then asked us to turn east. I told them we 
would rather not because of the weather and we were given present heading and to maintain 
3000 ft. The captain descended to 3000 ft. and kept going to 2500 ft. even though I told 
him our altitude was 3000 ft. His comment was, “You just look out the damn window.” 


This last example illustrates the tragic consequences of a captain from the other 
extreme—one who would not make a decision when one was required (NTSB, 1982; 
Burrows, 1982; Foushee, 1984): 


“Slushy runway. Do you want me to do anything special for it or just go for it?” asked the 
First Officer of Air Florida’s Flight 90, as he peered into a snowstorm at Washington 
National Airport... 


“Unless you got anything special you'd like to do,” quipped the plane’s 34-year-old captain. 
Shortly after brake release, the first officer expressed concern with engine instrument readings 
or throttle setting. Four times during takeoff roll he remarked that something was “not right,” 
but the captain took no action to reject the takeoff. (Air Florida operating procedures state that 
the captain alone makes the decision to reject.) 


Seconds later, Flight 90 came back down, hitting the 14th Street Bridge before it crashed into 
the ice covered Potomac River, killing 74 persons on the aircraft and four motorists on the 
bridge. 


The NTSB ruled that the captain of the aircraft did not react to the copilot’s repeated, subtle 
advisories that all was not normal during the takeoff. Moreover, in recommending that pilot 
training include “considerations for command decision, resource management, role 
performance, and assertiveness,” the Board implied that the copilot’s lack of assertiveness 


(possibly induced by the inherent role structure of the cockpit) may have been a causal factor. 
(NTSB, 1982, pp. 67—68) 


It is obvious that some crews do not do as well as they should. Yet in the course of 
our research on crews we have seen evidence of crews that go well beyond the call of 
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duty—crews that do better than the collection of individual skills available to them. For 
a truly remarkable account of a leader and crew “flying an un-flyable aircraft,” readers 
should review the NTSB account of United flight 232 captained by Al Haynes, which 
will be briefly described later in this chapter. If we are to understand effective crew 
performance, it is essential that we move beyond our focus on the individual to a broader 
level. We must begin to pay serious attention to the crew as a group if we are to optimize 
cockpit resources. 


3.2. Crews, Groups and Teams 


Groups fly crew-served airplanes, for a number of reasons. “As a direct result of the 
limitations and imperfections of individual humans, multi-piloted aircraft cockpits were 
designed to ensure needed redundancy” (Foushee, 1984). Furthermore, the Federal 
Aviation Regulations require at least a second in command if the aircraft is designed to 
carry more than ten passengers (FAR 135.99). At a minimum then, commercial flights 
will have a dyad (the smallest group) in the cockpit. The other extreme observed in our 
research was a crew of 25 aboard a military C-5 Galaxy. Whether a dyad, a triad, or 

a crew of 25, these are all groups and as such share the potential strengths and weaknesses 
that are inherent in groups. 

As illustrated earlier, groups are something more than merely a collection of the 
individuals comprising them. Some groups do remarkably well with no particularly 
outstanding individuals. Other groups, made up almost exclusively of high- 
performing individuals, do not do at all well as a team. A review of the performance 
of some of the US Olympic teams illustrates this phenomenon quite well. The 1988 
US Olympic basketball team is remembered, if at all, for not winning the gold 
medal. Yet the team had high-performing individuals, many of whom went on to 
play in the National Basketball Association, and the coach was highly respected. How 
could this happen, many asked? In the view of color commentator and former coach 
Al McGuire, the problem was that they did not have a “team,” but merely 
a collection of high-performance individuals. As McGuire recalls, the USA had 
a history of putting together basketball teams by selecting the best individuals 
available but doing little to foster or coordinate teamwork. In previous Olympics, 
when our individuals were much superior to the rest of the world’s individual 
players, we could win in spite of our lack of true teamwork. But as the rest of the 
world improved, particularly in the work of their teams as a whole, individual ability 
could no longer do the job. In what may be McGuire’s most remembered quote, he 
said ““You have to remember, there’s no ‘T’ in team.” He also noted that if we want to 
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win, we have to quit building “all-star teams” and instead build a team. “Team” is 
precisely what coach Mike Krzyzewski focused on with his US basketball team in the 
2008 Olympic Games in China. 

Perhaps an even more extreme example was the famous 1980 US Olympic hockey 
team, which is remembered for “the impossible dream come true” as it beat the Soviet 
team. Here was a team of no overwhelmingly great individuals—but a great team. They 
practiced over 100 games together as a team. Rather than being rewarded solely as 
individuals for goals, assists, saves, and the like, they were rewarded for the play of their 
lines (the five-man sub-groups that take the ice together) and for the performance of the 
team as a whole. They learned to work as a team and found that a team can overcome 
individual inadequacies, deficiencies and errors. 

Although these are excellent examples of team performance (or lack thereof), 
one does not have to go to the intense level of Olympic competition to 
demonstrate the same phenomenon. A technique used widely in helping groups to 
understand the value added from team performance is a classroom exercise designed 
to demonstrate synergy. In this exercise, individuals are presented with 
a hypothetical scenario which places them in an uncommon setting and asks them 
to rank order a limited number of items critical to their ultimate survival. While 
the specific task can vary widely (from “Lost on the Moon” to “Desert Survival”), 
the procedures remain common. After the individuals have completed their own 
rank orderings, they are placed in a group which represents the other survivors in 
this unique setting. The group’s task is to arrive at a consensus rank ordering of the 
same set of critical items. Upon completion of the rankings, both the individual 
and group rankings are compared to an ordering by experts in the particular setting 
(e.g. desert survival experts). Regardless of the specific nature of the setting, the 
results are virtually always the same (Kerr & Tindale, 2004). The most common 
result is that all of the groups’ performances will exceed the performance of any 
individual in any group. 

The parallel between lessons learned from this exercise and those learned in many 
aircraft accidents is more than casual. The characteristic of the classroom task that 
results in such predictable outcomes is its high degree of ambiguity to the participants. 
None of us has been lost on the moon, and it is such a unique environment that our 
experiences as individuals here on Earth are not particularly useful. Only when we 
integrate a number of varied experiences are we likely to arrive at a high-quality 
solution. Similarly, we seldom crash airplanes when we know exactly what the problem 
is and how to handle it. Even with major problems in critical periods of flight (such as 
loss of one engine at Vr), we are trained to handle them. In many accidents in today’s 
complex systems and environment, it is common to find that some aspect of the 
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environment or situation created ambiguity which, by definition, eliminated 

a structured solution. After all, if you do not know what the problem is, it is unlikely 
that you know what the solution is! But if you can get two or three independent 
critical thinkers involved, you will have a better chance of ruling out individual biases 
and will be on the road to a more effective solution. It is important to acknowledge, 
even in these hypothetical examples, that there must be time available to have effective 
group work. As I note in a later section, time-critical emergencies drive different 
strategies. 

In order to better understand group behavior and the impact of the group on the 
individual, it is necessary to become familiar with conditions that are uniquely 
associated with groups themselves. These are characteristics that can either only be 
defined relative to the group or, if associated with individuals, only make sense in 
a group setting. 


3.2.1. Boundaries 


Boundaries for a group are like the fence around a piece of property. A group boundary 
allows us to know who is in the group and who is not, whether or not we are a member 
of the group. It defines both physically and psychologically who the members of the 
group might rely on within their own group’s boundaries and thus indicates when it may 
be necessary to go beyond their own group for assistance or resources. A cockpit crew 
has a number of members defined by the design of the aircraft. A Boeing 757 has seats for 
two cockpit members, and hence there is an expected boundary of two for the crew of 
that airplane.’ A psychosocial boundary might also define the limits of tolerable deviance 
for group members. For example, all the types of socially acceptable and unacceptable 
behaviors are never made absolutely clear and are seldom written down. Thus, if a group 
can identify a boundary maintainer (usually someone close to the edge of acceptable 
behavior), they will have some means of gauging their own behavior as to its 
acceptability. 





"Sometimes technology overtakes original design. For example, on a C-141 aircraft 
there is a seat and workstation for a navigator. But the incorporation of inertial 
navigation systems and GPS has eliminated the requirement for the navigator 
position. Interestingly, this crew restructuring has also changed the social dynamics 
of the crew as predicted by sociotechnical systems theory, but that is another story 
for another chapter. 
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3.2.2. Roles 


A role is a set of expected behaviors associated with a particular position (not person) in 
a group or team. In any group setting over time, various roles will emerge. Some people 
will assume roles that are focused on accomplishing the task while others will take on 
behaviors associated with maintaining relationships within the group. Still others may 
take on roles that are counterproductive or even destructive to the group. Examples of 
some of the group roles which have been identified are listed in Table 3.1. 

Airline crews have clearly defined roles for the most part. The captain is the leader of 
the crew, followed by the first officer and second officer in turn. The lead flight attendant 
occupies a similar leadership position for the flight attendants. Some aspects of these roles 
are defined by law. Federal Aviation Regulation 91.3 states, “The pilot in command (i.e. 
the captain of a commercial aircraft requiring more than one pilot) of an aircraft is directly 
responsible for, and is the final authority as to, the operation of that aircraft.” Other role 
expectations are defined by the organization, or even by the crew itself. 

To the extent roles are clear and independent, the group will tend to function well, at 
least from a role standpoint. However, there can be role problems which will cause stress 
for the individuals involved and typically decreased performance from the group. Two 
kinds of role problems are most common. 


Role conflict 


When the individual is getting contradictory messages or expectations about his or her 
behavior, he or she is experiencing role conflict. These conflicts can come from several 
different sources. Perhaps most common is where the person is receiving two different 
signals about the expectations for a particular role. We can attach a label to this kind of 
role conflict depending upon from whom the signals are emanating. If the same person is 
giving you conflicting signals, we call that intra-sender role conflict. (“I want you to do 
a high-quality, detailed job and I need it in two minutes.”) If two different people are 

providing differing expectations about your role, that is labeled inter-sender role 


Table 3.1 Commonly identified group roles. 


Task roles Maintenance roles Blocking roles 


Initiator contributor Harmonizer Dominator 





Information seeker Encourager Blocker 


Information giver Gatekeeper Aggressor 





Evaluator Compromiser Disruptor 





Summarizer 


Crews as Groups: Their Formation and their Leadership - Chapter 3 


conflict. Sometimes the conflict can be between two different roles held by the same 
person. For example, a newly upgraded first officer may have conflicts between his role 
as father and Little League baseball coach and his flying schedule, which is now based on 
low seniority. This is referred to as inter-role conflict. Last is the situation where the 
expectations of a role violate the role occupant’s personal expectations or values. This is 
known as person/role conflict. Person/role conflict can also develop as the expected role 
migrates from initial expectations. An extreme example might be a person who was 
recruited by an intelligence agency to conduct analysis and, through a series of 
unexpected changes, is asked to engage in covert operations. 


Role ambiguity 


In role conflict, one receives clear messages about expectations but the messages are not 
all congruent. In situations of role ambiguity, the problem is that one cannot be sure 
what the expectations are at all. The information about the role is either lacking or not 
clearly communicated. Role ambiguity is more apt to occur in management positions 
than in traditional cockpit crew roles. 


3.2.3. Norms 


Norms are the informal rules that groups adopt to regulate group members’ behaviors. 
Although these norms are infrequently written down or openly discussed, they often 
have a powerful, and consistent, influence on group members’ behavior (Hackman, 
1976). One might reasonably ask, “if norms are powerful (so they are something I need 
to know about) but they aren’t written down and aren’t discussed, how am I supposed to 
figure them out?” Fortunately, most of us are rather good at reading the social cues that 
inform us of existing norms. When we first enter a work situation, even though there 
may not be a dress code, we are fairly astute at determining that “everybody around here 
wears a suit.” We also are apt to notice a norm if it is violated, even though we may have 
been unable to articulate the norm before its violation was apparent (e.g. the guy 
wearing jeans when everybody else is wearing a suit). 

Another fortunate aspect of norms is that they do not govern all behaviors, just those 
behaviors that the group feels are important. Feldman (1984) has outlined four reasons 
why norms are likely to be enforced. He suggests norms are more apt to be enforced if they 
(1) facilitate group survival; (2) simplify, or make more predictable, what behavior is 
expected of group members; (3) help the group avoid embarrassing interpersonal 
problems; or (4) express the central values of the group and clarify what is distinctive about 
the group’s identity. 
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An outsider is often able to learn more about norms than an insider for several 
reasons. First, the outsider (not necessarily being subject to the norms herself) is more 
apt to notice them. In fact, the more “foreign” the observer is, the more likely the norms 
are to be perceived. If one is accustomed to wearing a tie to work, one is less likely to 
notice that another organization also wears ties to work, but more likely to note that 
a third organization typically wears sweaters and sweatshirts around the office. Another 
lesson the outsider can learn by observing other groups’ norms is something about his or 
her own group’s norms. In a recent consulting project, our research team was struck by 
the failure of the client organization to share information with us—not proprietary 
information, but information that impacted our own ability to work with them. In 
a moment of reflection on this situation, we realized that our work group norm was very 
different from theirs. Our team had a norm that encouraged open sharing of 
information with each other—but prior to seeing a very different norm in a different 
group, none of us could have articulated our own norm of openly sharing information. 


3.2.4. Status 


Status is the relative ranking of individuals within a group setting. In an airline cockpit 
crew, status is typically associated with the roles of captain, first officer and, if 
appropriate, second officer. In these cases, status comes with the position. Status, like 
roles, determines appropriate behaviors for all group members. Usually a high-status 
person has more power and influence, and thus the lower-status members of a group 
tend to defer to the higher-status members. Again, crossing cultures gives us interesting 
insights into status impact. In Eastern cultures, age is given status and younger people 
will bow to older people. Since Western culture lacks castes or clear-cut status lines, it is 
sometimes difficult to figure out who has the most status. Status incongruence can result 
in stress for the individuals and less than satisfactory work outcomes. Tom Wolfe in The 
Right Stuff (1979) describes the status incongruence that occurred between the flight 
surgeons (who believed they were the most important people in the manned space flight 
program—after all, they could reject an unfit “subject” with the stroke of a pen), and the 
test pilots who were to become the astronauts (who believed they were the very reason 
there was a manned space flight program). 


3.2.5. Authority 


Technically, authority is the right to use power and influence. People derive authority in 
the group setting from the legitimate power given them by the organization. The 
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captain has the authority to order a drunken or abusive passenger off the airplane or to 
not accept a flight that he or she believes is unsafe. Authority can also be granted on the 
basis of recognized expertise or expert power. Again, the group can get into trouble 
when differing sources of authority clash. There have been numerous reports of 
accidents caused by confused authority dynamics in the cockpit. For example, several 
accidents have occurred in military cockpits when a higher ranking (status) officer was 
assigned as a check pilot for a junior ranking crew and then became involved in giving 
directions during an actual emergency. The confused authority dynamics were directly 
responsible for accidents of this nature. Such confusion was also possible in the case of 
the senior captain who was forced to retire at age 60 by FAA rules but then decides to 
assume the position of flight engineer. Even though that former captain may be entirely 
clear on the limits of his authority in his own mind, his former status may create 
confusing authority dynamics for his younger crewmates. 

Authority dynamics have their roots in the dependency relationships we have 
developed from birth. As children, we were dependent on our parents and accepted their 
authority. As we grew and became more independent, we had to work through the 
evolving authority relationships. Even today, we are all dependent at certain times. 
Passengers in commercial aircraft are dependent on the crew. A “dead-heading”’ first 
officer with 10,000 hours of flying time is still dependent on the crew flying in the 
cockpit. There is nothing good or bad about being dependent unless we mismatch the 
degree of dependency and the situation. A passenger who decides to take over the airplane 
has inappropriately usurped authority. At the other extreme, a first officer who becomes 
overly dependent on the captain for decision-making is not likely to help the crew either. 
Yet authority dynamics can result in just such occurrences. In an investigation conducted 
by Harper et al. (1971) at a major air carrier, captains feigned incapacitation at 
a predetermined point during final approach in simulator trials characterized by poor 
weather and visibility. In that study, approximately 25% of these simulated flights “hit the 
ground” because, for some reason, the first officers did not take control even when they 
knew the plane was well below glide slope. We can assume from this research and from 
other artifacts (see below) that the authority dynamic surrounding the role of the captain 
must be extremely powerful. Figure 3.1, which depicts a sign found on a bulletin board in 
a commercial carrier’s operations room, is only partly facetious. 


3.2.6. Group Dynamics 


Clearly, all the topics in this section on groups could fall under the general heading of 
group dynamics, since they are all dynamic characteristics that only occur in a group 
setting. Recognizing the confusing nature of groups themselves, especially in our 
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Figure 3.1 Sign Posted on an airline’s bulletin board 















THE TWO RULES OF 
COMMERCIAL AVIATION 


Rule 1: The Captain is ALWAYS right. 
Rule 2: See Rule 1. 













culture, it seems best to discuss a few group dynamics topics separately. There are, of 
course, many more group topics than I have space to discuss here. However, in closing 
we should note two remaining dynamics of groups. 

Social influence is a by-product of group activity. Unfortunately, it has both positive 
and negative components. On the positive side is an effect labeled “social facilitation.” 
This construct suggests that, in general, people are aroused by the presence of others and 
more motivated to perform well, knowing that others are evaluating them. On the dark 
side of social influence is what Latane et al. (1979) have called “social loafing.” Here the 
individual members of the group feel less pressure to perform when they are working 
with others. The researchers believe this may happen when the individuals are only 
carrying part of the load and no one can tell which member is loafing. 

Groupthink is another flaw of highly cohesive groups, discovered by Janis (1982). He 
found that when people became deeply involved in a highly cohesive group, they often 
became more concerned with striving for unanimity than in realistically appraising 
alternative courses of action. This condition can be exacerbated when the leader 
promotes his or her preferred solution and when the group is insulated from expert 
opinions outside the group. Janis believed that groupthink accounted for a number of 
historic fiascos, including the US’ failure to heed warnings of the impending attack on 
Pearl Harbor, the decision processes leading up to the failed Bay of Pigs invasion, and the 
Watergate cover-up. 


3.3. Group Process and Leverage 


Having briefly discussed some of the characteristics associated with groups, teams and 
crews, we may now begin to consider a model for improving their output. Merely, 
the mention of the word “output” leads us to begin thinking in the language and 
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models of systems theory with its familiar terminology of “input-process-output.” 
While that concept may be useful for considering group work, interventions or 
corrections based on systems theory have not been too successful. In systems theory, 
inputs are generally “givens” and outputs are “desired.” If the outputs are not meeting 
expectations, then the corrective intervention most typically occurs somewhere in the 
“process” stage of the system. Since the 1970s, much of our group-oriented corrective 
interventions have pursued this course of action by attempting to intervene in the 
process stage of the group’s work (see Schein, 1969, for a discussion of process 
interventions). After all, that is where the problems were most obvious—why not fix 
them where you see them? Unfortunately, years of evidence did not support that 
concept (Kaplan, 1979). That does not mean that process interventions cannot be 
helpful, but they should not be expected to fix all the problems encountered by 
groups either. If one buys an extremely cheap automobile, no amount of work by 
a mechanic will make it perform and ride like a Mercedes-Benz. Some things are far 
better incorporated in the design (input) phase than in the maintenance (process) 
phase. 

Hackman (1987) and Ginnett (Hughes et al., 2009) have proposed models to design 
groups for output effectiveness. Their models suggest that the organization should be set 
up to support group work and also that the group should be designed to accomplish 
output objectives. Two important points should be noted in these models. First, the 
output is not unidimensional—it is not exclusively focused on satisfying the 
organizational or client needs. Certainly, that is an important consideration, but both 
Hackman and Ginnett also note that the group must be able to continue to perform in 
the future, and the individuals making up the group should obtain at least as much 
satisfaction as dissatisfaction from working in the group. For example, if a cockpit crew 
flies a “safe and efficient” leg in a trip, that would meet the first criterion. But if, in the 
process of the trip, there was so much interpersonal tension that the crew felt they could 
no longer work together on subsequent legs, the output of the group would not be 
labeled as effective. 

Most organizations (airlines included) cannot afford to wait until their teams 
disintegrate or fail to perform their required tasks successfully before taking corrective 
action. This is where process criteria can be helpful, not as points for intervention but as 
points for diagnosis. By paying attention to how the group is going about its work, we 
may infer that their ultimate performance may have problems as well. But rather than 
intervening first at the process level, it makes more sense to use leverage at the input 
level. Ginnett’s model discusses factors at the organizational, team or group level and at 
the individual level which can support group-level work. It is at the team or group level 
of leverage that we will focus our attention. 
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3.4. Leadership 


Having just stated the focus to be group design and then labeling this section 
“leadership” might trigger a few questions if not alarms. Two such questions might be: 
(1) How can we do anything about group design—aren’t cockpit crews based on the 
design of the cockpit? and (2) What does leadership have to do with groups—I thought 
leadership was about leaders? 

Let me address the second question first. Leadership is about leaders. But it is not 
about leaders in a vacuum—it is about leaders in relation to followers in a particular 
setting. Is there such a thing as leadership without followers? And since we have already 
agreed that any two people comprise a group, if there is a leader and at least one follower, 
we are in the group realm. The fact is leadership is a group phenomenon. 

This contributes directly to our answer to the first question. Anyone who has spent 
much time watching groups operate in organizational environments will tell you that 
they do not all work equally well. Some cockpit crews cause accidents (as we have already 
noted), yet other cockpit crews exceed our greatest expectations. As one example, 
Captain Al Haynes and the crew of United 232 en route from Denver to Chicago 
suddenly found themselves in a situation that was never supposed to happen. After 
a catastrophic failure of the DC-10’s number 2 engine fan disabled all three hydraulic 
systems, this crew was left with little or no flight controls. Captain Haynes enlisted the 
assistance of another captain traveling in the passenger cabin and, with his newly 
expanded crew, literally developed their own emergency procedures on line. In the midst 
of crisis the crew of United 232 managed to get the crippled airliner within a few feet of 
the Sioux City airport before impact. Remarkably, this crew performed even better than 
subsequent crews in simulator re-enactments—even when those crews were comprised 
of test pilots. If some crews work better than others in the same organizational setting, 
then something about those crews must be different, and it must have something to do 
with the design of the groups. For airline crews, this “crew design” begins to occur when 
the crew first forms. But what is responsible for the difference? 

In numerous interviews with crewmembers about this variation among crews, the 
same consistent answer emerged. Whether a crew works well or not is a function of the 
captain. One typical example of interviews of subordinate air crewmembers conducted 
by this author (Ginnett, 1987) illustrates this point: 


RCG: Are all the [captains] you fly with pretty much the same? 


PILOT: Oh no. Some guys are just the greatest in the world to fly with. I mean they may 
not have the greatest hands in the world but that doesn’t matter. When you fly with them, 
you feel like you want to do everything you can to work together to get the job done. You 
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really want to do a good job for them. Some other guys are just the opposite... you just can’t 
stand to work with them. That doesn’t mean you’ll do anything that’s unsafe or dangerous 
but you won’t go out of your way to keep him out of trouble either. So you’ll just sit back and 
do what you have to and just hope that he screws up. 


RCG: How can you tell which kind of guy you’re working with? 
PILOT: Oh, you can tell. 
RCG: How? 


PILOT: I don’t know how you tell but it doesn’t take long. Just a couple of minutes and 
you’ll know. 


Not only does this illustrate the perception of the impact of the leader (the captain), 
but it also points to the critical nature of the crew formation (i.e. “Just a couple of 
minutes and you'll know’’). 

The pervasive impact of the leader has been demonstrated in controlled research 
settings as well. I have already cited the feigned incapacitation study by Harper et al. 
(1971), in which the authority dynamics associated with the captain’s role impacted the 
performance of the first officers. In another simulator study, Ruffell Smith (1979) 
designed an experiment where crews were given an interactive problem soon after 
departing on an intercontinental flight. The problem required a return to a short, wet 
runway with a number of interrelated mechanical problems and a critical fuel dump. The 
workload burden fell on the engineer, so the most obvious predictions about which crews 
would be able to safely return centered around the engineer’s performance. A very 
detailed analysis of the number and type of errors showed great variations among the 
crews. As it turned out, the variable of most significance was not the flight engineer’s 
behavior but the behavior of the captain. If the captain recognized the problem as a crew 
problem and managed the problem accordingly, the crew did well. However, if the 
captain handled the problem as “a piloting problem,” the crews did not fare as well. 
Apparently, the captain’s behavior carries considerable weight in the way the crew works. 
And if the interview data are valid, the leadership impact begins early in the crew’s life. 


3.5. Leadership at Formation: A Critical 
Leverage Point 


The first phase of our NASA research set out to address the question of what actually goes 
on in the formation process of cockpit crews (Ginnett, 1987). Of particular interest was 
the behavior of captains who, prior to observation, were assessed by check airmen as 
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being exceedingly good at creating highly effective teams (the HI-E captains) versus their 
counterparts who received low ratings on this same ability (LO-E captains). In 
accordance with accepted research procedures, the category of the captain to be observed 
was not revealed until after all data collection and content analyses were completed. 

It may be helpful to briefly explain the context within which the first phase of the 
research occurred. Phase One was conducted entirely with crews assigned to 727-200 
aircraft so the technology, crew size and training were standardized. The particular 
airline company in which these first data were collected used a fairly typical bid system 
for crew scheduling. As a result, the crews were quite likely never to have worked 
together prior to coming together for an assigned trip. Of the 20 different three-person 
crews observed in the first phase of the research, none had ever worked together prior to 
the observation period. In fact, of the 60 dyads within the 20 crews, only eight had ever 
flown together before, and seven of those eight had done so only once. Their operations 
manual required a formal crew briefing before the first leg of each new crew 
complement. This briefing, conducted one hour before scheduled departure, was held 
in a designated room in the terminal unless there were late arrivals, in which case the 
briefing would occur on the aircraft. It is important to note that whether an organization 
requires a formal briefing or not (as was the case in subsequent organizations researched), 
there will be a crew formation process. If the organization does not legitimize this 
process with a required briefing, then whether the formation process occurs by design or 
by chance is very much up to the captain. 

Based on extensions of the normative model by Hackman and Walton (1986) and 
observations of team formations in organizations other than airlines, I had certain 
expectations of what effective leaders would do when forming a team that had never 
worked together before. It seemed reasonable to expect a team leader to: 


1. Discuss the task to be accomplished by the group. 


2. Discuss the relevant team boundaries. Since this was a team that had never worked 
together before, I expected the leader to build a tight-knit working group. 


3. Discuss relevant norms for the group’s effective performance. 


There were some surprises in what I found. 


3.5.1. Task Findings 


Contrary to expectations, the HI-E captains hardly discussed tasks at all. 
Even when tasks were mentioned (e.g. closing the cabin door, retracting the aft air 
stairs, or keeping the cockpit door open prior to pushback), they were more about 
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boundary issues (to be discussed in the next section) than about the tasks themselves. 
The only other exceptions which generated some task discussion occurred when there 
were unusual conditions such as weather or performance limitations due to deferred 
maintenance items on the aircraft. In contrast, some of the LO-E captains spent 
inordinate amounts of time discussing minute task requirements for the flight attendants 
which had little to do with boundary requirements or any other critical aspect of team 
performance. One LO-E captain went into great detail about procedures for bagging the 
cabin garbage! 

But the general absence of task discussion was far from the predicted behavior—or 
from behavior exhibited by leaders in other task groups. For example, in problem- 
solving groups (often assembled in organizational settings as ad hoc committees), the 
bulk of the first meeting is spent defining and clarifying the task at hand. How can we 
explain the lack of task discussion by HI-E captains, and in sharp contrast, the focus on 
even trivial tasks by LO-E captains? 


3.5.2. Boundary Findings 


As noted earlier, it might appear that an airline cockpit crew, or even the total crew 
including the attendants, is a fairly well-defined and bounded group. After all, when you 
seal a work team in a pressurized aluminum cabin at 35,000 feet, there is little chance of 
someone leaving the group. In fact, based on the behaviors of the HI-E captains, they 
felt the groups were potentially overbounded. The HI-E captains worked both in the 
briefing and at other opportunistic times to expand the relevant team boundary and to 
make the boundary more permeable. They always talked about “we” in terms of the 
total flight crew, as opposed to some of the LO-E captains who referred to the cockpit 
crew as “we” and the flight attendants as “you.” The HI-E captains also worked to create 
a larger vision of the relevant work group—one that exceeded the bounds of the aircraft. 
They took pains to include (at least psychologically) gate personnel, maintenance and air 
traffic controllers as part of the group trying to help them, not as an outside hostile group 
trying to thwart their objectives. One HI-E captain routinely reminded the crew that 
the passengers could be a relevant part of their team if the crew made the effort to listen 
to passengers, particularly if they were expressing some concern about the aircraft. 


3.5.3. Norms Findings 


Norms can be communicated in a variety of ways. Certainly, the captain can make 
explicit the standards and expected behaviors of the crew. She can communicate the 
importance of a subject merely by including it in the briefing, or he can talk explicitly 
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about its importance. The captain can also communicate normative information 
through a modeling process. This may include specific descriptions of intended 
behaviors or, more subtly, be expressed through actual behavior in the briefing and at 
other times in the presence of the crew. For example, a captain may quite subtly transmit 
the importance of exchanging information as the group goes about its work by merely 
taking time to exchange information (two-way communication) in the time allotted for 
the crew briefing. The norm that “communication is important” is expressed in the 
series of exchanges including: (1) I need to talk to you; (2) I listen to you; (3) I need you 
to talk to me; or even (4) I expect you to talk to me. 

There was no single norm that was explicitly communicated by all of the highly 
effective captains. However, there were three norms most frequently communicated as 
important to the effective work of the group. These were the importance of safety, 
effective communication and cooperation between crewmembers. Perhaps most 
surprising is that “safety” should need to be mentioned at all! Is that not the most 
important consideration anyway? That safety should be emphasized also seems to be 
contrary to the finding regarding tasks which were not mentioned much at all by the 
HI-E captains. These apparently conflicting and confounding findings are explained 
later in the section on “group shells.” 


3.5.4. Authority Dynamics Findings 


While not a factor outlined in Hackman & Walton (1986) as something to which the 
leader should attend in group formation, the authority dynamic was such a powerful 
finding that it could not be overlooked. Certainly, the use of influence and authority are 
common issues in leadership writings as far back as Lewin et al. (1939) and often are an 
integral part of leadership definitions. To understand the authority dynamics for airline 
cockpit crews it will first be necessary to provide a small amount of background 
information. The authority relationship between the captain and the rest of the crew is 
inexorably bound to aviation history, regulations and often to the characteristics of the 
crewmembers themselves. This combination of history, regulation and crewmember 
characteristics has established an authority dynamic that has undoubtedly positively 
impacted the aviation safety record. In those situations requiring immediate response to 
a single authoritative command, airline crews work particularly well. However, this 
tendency toward the high-authority end of the continuum has resulted in crewmembers 
not speaking up when necessary, as in the previously cited study and accident 
investigation. This inclination may also result in excessive psychological dependence on 
the captain as leader to the extent that individual contributions to problem-solving are 
neither voiced nor attempted. For example, one captain with whom I flew made 


Crews as Groups: Their Formation and their Leadership - Chapter 3 


a particularly poor approach which resulted in an excessive dive on short final, thus 
setting off numerous alarms. In reviewing the crewmembers’ inactions afterward, the 
young second officer (who literally said nothing during the final approach) admitted that 
he had never seen an approach quite like that, but figured “the captain must know what 
he’s doing.” 

If we plot authority dynamics along a continuum (as opposed to Lewin et al., 1939, 
who used nominal categories), the history, regulations and individual characteristics of 
crewmembers all tend to be forces pushing toward the high end of authority use and 
response (Figure 3.2). 

As noted above, there are occasions in aviation where the extreme high end is 
appropriate, and most of us would agree that we cannot afford (nor do we personally 
want) the low end to occur. Given the existing history, regulations and backgrounds, the 
latter condition is unlikely to occur. In fact, if we exclude hijackings and suicides, 

a review of the records of aviation accidents cannot produce a single incidence of 
“accident due to mutiny.” 


Establishing appropriate authority 


One might expect HI-E captains to deliberately move the authority dynamic back down 
from its pre-existing extreme point to a level more appropriate for group-level work 
(i.e. somewhere in the middle of the continuum). Under such a hypothesis, the leader 
might operate solely in a more democratic or participative fashion. Such a finding 
would be simple and prescriptive. Unfortunately, that simplistic approach is not what 
happens. 

Rather than operating at some specific point between complete democracy and 
complete autocratic behavior, the highly effective captains shifted their behavior during 
the formation process all along the continuum between the extremes of the effective 


Figure 3.2 Range of authority dynamics in crew work 
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range. Again, note that the highly effective captains never exhibited laissez-faire 
behaviors. Three methods were used to build an effective leader/team authority 
relationship: (1) establish competence; (2) disavow perfection; and (3) engage the crew. 


1. 


Establish competence. In addition to the other statements made by the HI-E captains 
during their briefings (like establishing norms for crew behavior), they demon- 
strated their capability to assume the legitimate authority given them in three 
ways. First, the briefing was organized along some logical parameter (e.g. 
temporal, criticality, etc.). This helped to establish competence by demonstrating 
the captain had given some thought to the work they were about to engage in and 
he or she was able to present this in an organized manner, thus indicating ratio- 
nality. Second, the briefing always contained elements of technical language 
specific to the vocation of flying. And finally, they were comfortable in a group 
setting—the environment of leadership. Like norms, this fact escaped recognition 
until its absence was observed among some of the LO-E captains. 


Disavowing perfection. All the HI-E captains established competence by exhibiting 
the above behaviors, but that only provided their crews with evidence that there 
was cause for the captain to exercise legitimate authority. Then, these captains 
balanced the leader/crew relationship by having the crewmembers take 
responsibility for the work of the group as well. This is important if the crew is 
not to completely rely on the captain, especially when he or she is in error. 

This was first noted in a captain’s statement prior to an extremely effective 
crew performance in a simulator: “I just want you guys to understand that 
they assign the seats in this airplane based on seniority, not on the basis of 
competence. So anything you can see or do that will help out, I’d sure 
appreciate hearing about it.” 

As simple as that sounds, it seems to underlie the basic behavior that HI-E 
captains use in disavowing perfection. They make a statement suggesting they 
don’t know something about a particular issue even though the information is 
often quite readily available. This is a delicate balance: they do not contradict the 
competence they have established regarding their ability as a captain. Rather, they 
typically make some comment about their lack of knowledge (although not on 
a critical task) or about some personal shortcoming. They are open about dealing 
with their own vulnerabilities. 


Engaging the crew. The HI-E captains became involved with and included the 
crews in the process of the briefing and in the social process of group formation. 
Content analysis of the briefing process showed specific instances where the 
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HI-E captains were engaging the crew through real-time interactions. They 
dealt with the situations that could potentially impact the particular crew 

they were briefing as they learned about them in the course of their interactions. 
They interacted on a personal level with the other people who were filling the 
crew roles. (For recent literature supporting the importance of leader/team 
interaction, see Kozlowski & Bell, 2003.) They did not present a “canned 
briefing,” nor did they provide a briefing that could just as well have been given 
to a group of mannequins. They interacted in the here-and-now with the other 
people with whom they would work. By dealing in real time with the people 
who were filling the roles, they conveyed important normative information 
about themselves and the value of the individuals who made up this particular 
group. They often did this with humor but it was not humor to isolate (canned 
jokes) but rather humorous responses to real-time interactions. 

The HI-E captains also spent more “non-directive” time with the group. It is 
not the case that these captains spent significantly more total time in the briefing 
with the crew than did the LO-E captains. Nor is it the case that they spent 
more time than the LO-E captains actually talking to the crew. There was, 
however, a significant difference between the HI-E captains and the LO-E 
captains in the amount of time that other members of the crew talked while the 
captain was present. The highly effective captains allowed and encouraged 
conversation by the other crewmembers, particularly if it was related to the task. 
They always asked if there were any questions, and several of them solicited 
comments about any behaviors on other crews or with other captains that might 
be troublesome. 

By establishing their competence, disavowing perfection and engaging the 
crew in the course of the briefing, the HI-E captains actually covered the range 
on the continuum of authority in which groups most effectively operate. Rather 
than demonstrate only one type of leadership authority which would be 
inappropriate across the range of requirements in a typical line operation (see 
Ginnett, 1990), these captains established, early on, an authority basis that would 
change according to the situation. This contingent authority pattern ranged 
from direct statements by a competent, legitimate authority figure to a human 
who recognized and was comfortable with his own imperfections. They further 
provided a mechanism for correcting these errors by ensuring that the crew was 
engaged and active in the task work already begun in the briefing. 


In summary, the HI-E captains did not dwell on the task, expanded the boundaries to 
include others who could help the group in its work, made explicit certain important 
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performance norms and created an expectation of flexible authority contingent upon 
the situation. 

What remains unresolved are (1) explanations of the unexpected or surprising 
findings concerning the absence of task discussion in contrast to the explicit discussion 
around norms associated with safety; (2) some understanding of how the leaders of these 
groups were able to accomplish the formation process so quickly; and (3) what the 
differing leadership behaviors had to do with subsequent performance. Fortunately, the 
concept of organizational shells” can help answer these questions. 


3.6. Group Shells 


The origin of group shells is similar in concept to shells in chemistry or shells in 
computer science. In chemistry, a shell is a space occupied by the electrons or protons 
and neutrons in an atomic structure. The shell can be qualitatively pictured as the region 
of space where there is a high probability of finding the particle of interest. Similarly, the 
organizational shell for a group will not guarantee that every component for its 
formation will be established. It merely suggests that somewhere within the bounds of 
the shell, one might expect to find certain behaviors, roles, norms, or dynamics 
occurring. 

In computer science, a shell provides a predefined set of interactions between various 
aspects of the system. Typically, these predefined sets of interactions occur between the 
computer and the operator. Analogously in organizational settings, a shell serves the 
same function—it provides a predefined or expected set of interactions between various 
elements of the system which permits simpler and more efficient interactions. With 
these two concepts as background, it is now possible to examine the data in light of the 
concept of the shell. 

The research described in part here was designed to examine the captain’s behavior 
during the formation process of crews in their organizational setting with all the relevant 
contextual information in place. This pre-existing context provides critical information 
for the forming group. Just as it was important for the reader to have some understanding 
of the relevant background of aviation-related authority dynamics to make sense of the 
findings in that area, so too is it important to recognize that all the task work described 





* The concept of organizational shells emerged in a working session between this 
author and Richard Hackman. Although we both remember that the concept first 
appeared “‘on the flip-chart on the back of the door,” neither of us recalls who used 
the term first. Hence, we agreed to jointly assume responsibility for the concept. 
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here occurs in an ongoing organizational and environmental context. The crews do not 
form in isolation but rather in an embedded system of intra-organizational, industry and 
environmental conditions (see Figure 3.3). 

One can see from this diagram that information critical to group work can come 
from a variety of sources and in varying amounts. For example, the environment and 
industry may provide a sufficient guarantee of capability such that the organization (or 
lower levels) need not expand upon these. In the case of the airlines, industry-level 
agencies such as the FAA and the Air Line Pilots Association provide minimum 
certification requirements for commercial pilots. Other requirements for effective 
group work may be left solely to the crew, and these elements may be added at the 
formation or other opportunistic moments later in the crew’s life. In light of this 
understanding of the concept of the shells, let us examine a few of the apparent 
anomalies in the data. 

How is it that HI-E captains forming their crews for the first time do not spend 
much time at all discussing the task? In contrast, since safety would seem to be the 
most important factor for commercial air travel (at least from the perspective of 
passengers), why is it that the HI-E captains do take time to discuss safety? And how is 
it that even the LO-E captains produce teams that, under normal conditions, perform 
satisfactorily? 

The answer to the first question lies in the nearly total fulfillment of task information 
from the shells outside of crew formation. All the individuals coming together to form 
the crew bring with them the knowledge, skill and training necessary to perform the 
group’s work. At increasingly redundant levels, the environment, the industry and the 
organization test and certify these abilities. Unlike a randomly selected group of college 
sophomores forming to complete a novel task in a social science laboratory, all these 
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crewmembers were highly qualified in the task requirements of a role that was designed 
to enable the group to work. Knowing that the outer shells have satisfied these task 
requirements, it would be extremely redundant for the leader to further discuss them. 
This is also consistent with the predictions of leader effectiveness according to path/goal 
theory as described by House and Mitchell (1974). In fact, when a LO-E captain spends 
time discussing obvious tasks, the crew begins to develop a very different picture of how 
life will be with him or her as their leader. 

But this explanation might seem to confound expectations regarding the time spent 
by the leaders in explicitly discussing safety. Certainly, the outer shells contain some 
normative expectations concerning safe operations. And if one were to ask any 
individual crewmember whether safety was important, it is reasonable to assume they 
would answer affirmatively. Then why spend time talking about a norm everyone 
accepts? The answer again is found in the shells, but in a more complex and ambiguous 
manner. Within the various shells there are numerous normative expectations for 
performance, among them safety. Unfortunately, not all the norms are congruent. 

A specific example will help to clarify this. 

Beyond the norm of safety which exists in all the shells, a highly supported norm 
from airline management (within the organizational shell) is fuel conservation. For 
a commercial carrier, fuel is typically the second highest expense, so anything that can be 
done to save fuel is reinforced. Thus, when takeoff delays are anticipated, captains will 
instruct their crews to delay starting all engines as a fuel conservation measure. This 
tactic has virtually no confounds with safety. But another fuel conservation technique 
might be to keep the airplane “as clean as possible for as long as possible.’ Pushing this 
technique to the extreme, a crew may delay extension of flaps and gear until late in the 
approach. The problem is that this practice might be in conflict with safety, which might 
prescribe an earlier and more gradual configuration for landing. By prioritizing 
potentially conflicting norms, the HI-E captains have clarified in advance their 
expectations, thus reducing ambiguity and potentially enhancing performance on the 
line. This will help the crew in routine operations and will be critical to effective 
performance in demanding or emergency situations. 

Lastly, the shells for airline crews provide sufficient structure to allow them to 
perform at some minimal level in spite of ineffective leader behavior. It is important to 
note in this context that we are not considering “optimal” group performance across 
normal line operations, but rather “satisficing” group performance (cf. Chapter 5 by 
Orasanu). This type of minimally acceptable behavior may well be less than necessary 
in demanding situations where crew resource management is essential. For a more 
in-depth discussion of the leader’s transformational behavior enhancing the safety 
behaviors in followers, see Barling et al. (2002). 
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It is critical to stress the importance of understanding the contribution made by the 
shells for the particular group under examination. This means that the particular findings 
from these groups should not be extrapolated directly to other groups unless their shells 
are similar. In these airline crews, the HI-E captains did not spend much time in the 
formation process dealing with the task because the task information was imported from 
the shells. However, in the first meetings of other groups (e.g. B-1 bomber crews on 
a new low-level night mission, or ad hoc task groups) it may be most appropriate for the 
leader to spend considerable time discussing the task to be performed, since the shells 
offer insufficient information about the group’s impending assignment. 

If we return to Hackman & Walton’s (1986) normative model, which suggests that 
the leader can make a contribution to the group at the critical formation period by 
discussing the task, the boundaries and the norms of the group, we may now be able to 
improve those prescriptions. First, authority dynamics must be added to the list. The 
leader needs to consider the pre-existing (shell-provided) authority issues and modify 
them in the direction of group effectiveness. For airline captains, the shell structure for 
authority was almost exclusively in the direction of the autocratic power of the leader. 
While that is sometimes appropriate, it may not be the best for effective group work, and 
so the leader should attempt to shift authority down the continuum while maintaining 
a contingency approach. Second, rather than suggesting the leader spend time discussing 
tasks, boundaries, norms and authority issues, it is more appropriate to say the leader 
should consider these issues and ensure information about them is provided in sufficient 
quantities for the group to get started and work effectively. The shells may provide all the 
necessary information for some groups and virtually none for others. In the former 
groups, discussion might be redundant, while in the latter case, discussion (in the 
absence of information) or clarification (in the event of conflicting information) may be 
the most important function the leader can perform at the group’s formation. Which 
behavior is most important can only be determined by understanding the data inherent 
in higher levels of the shells. 


3.7. Implications for Effective Crew 
Leadership 


From the research described here, it should be fairly obvious that the captain can make 
a difference. Assuming we have an organizational context that supports and sustains 
crew and team effectiveness, the captain has available to him or her the critical period 
of crew formation. This is where the captain breathes life into the shell which is filling 
with others who will play predefined roles. How well or how poorly the crew 
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performs is, in large part, established in the course of the first meeting (Weick, 1985; 
Ginnett, 1987). 

I have already detailed four specific areas in which the captain can create effective 
conditions for crew work. Beyond this are four more general categories which describe 
the captain’s overall response to the shells at the group level. 


3.7.1. Undermining 


A captain who “undermines” actively countermands the conditions inherent in the shell 
that each member imports to the crew situation. These are the captains who, through 
their behaviors (including explicit statements), redefine in a more restrictive and 
unconstructive manner the tasks, boundaries, norms and authority dynamics which will 
guide the crew’s operations. These captains create conditions that undermine crew 
effectiveness. In an organization with established shells that foster effective crew work, 
undermining captains negate the pre-existing and positive shells. Not only can they 
reduce and restrict positive aspects of the shell by explicitly undermining them, but 
their general tendency to undermine is extrapolated to other areas of the shell they do 
not mention. If a captain says he does not want flight attendants to get off the aircraft to 
talk to gate personnel without his permission, the flight engineer who overhears this 
may well wonder whether he or she needs the captain’s specific approval to conduct 
a walk-around inspection of the aircraft. Worse yet, should he or she take the initiative 
to plan ahead for the crew’s benefit or wait to see if it is “what the captain wants?” If 
captains go against procedures on one aspect of performance, what can they be 
expected to do on others? The most widespread negative effect of undermining 
behavior is that, like a cancer, it metastasizes throughout the organization. 
Unfortunately, the reduced shells that result from interaction with an undermining 
captain may be subsequently imported to other crews with the same potentially negative 
impact. If a captain can behave inappropriately (as defined by existing organizational 
shells) and the organization fails to correct that inappropriate behavior, the other 
members of the crew will doubt the validity of the shells and hence expect less of 
subsequent captains and crews. 


3.7.2. Abdicating 


Captains who “abdicate” neither confirm the pre-existing shell nor deny it. 

They add nothing to the shell, nor do they confirm what the environment and 
organization have put in place. Crews under these kinds of captains are “not sure” —they 
are left with whatever shell they arrived with, minus any confirmation of its current 
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utility or appropriateness. Not only is the shell for this particular crew left unverified, but 
each crewmember’s shell used for defining the role of “captain” is reduced because of 
this particular captain’s performance. They leave with a “less clearly defined and 
potentially poorer” shell of what the organization expects of its captains. This is because 
it is very likely that the organization, if not the environment, has authorized the captain 
to clarify and even modify the shell, and this captain has failed to do that. Therefore, 
extrapolations regarding his or her self-imposed diminished authority in a more general 
sense are apt to be the result. By abdicating, the captain has unwittingly exhibited some 
of the behaviors inherent in the previous category. 


3.7.3. Affirming 


At a minimum for crew effectiveness, the captain should affirm the constructive task 
definitions, boundary conditions, norms and authority dynamics that the environment 
and the organization have structured into the shell. These behaviors would not expand 
the shell but would help solidify the crew’s understanding and acceptance of it. In 
effect, each crewmember arrives with a shell that has generally defined appropriate 
crew behaviors in the past. The “affirming” captain “fills in the existing dotted lines” 
so the crew can proceed with behaviors based on their imported expectations. To 
the extent the organization and the environment have provided a shell appropriate 
for crew effectiveness, the crew under an affirming captain can be expected to 
perform well. 


3.7.4. Elaborating and Expanding 


These are the behaviors of the best leaders. They appreciate and exploit the 
opportunity for crew effectiveness provided them at the time of crew formation. They 
expand the existing shell and create new ways to operate within and outside of its 
boundaries. These are the leaders who expand and create new opportunities for 
constructive interactions among crewmembers. They tend to elaborate and enlarge the 
boundaries of the individual roles and of the crew as a whole. They also create semi- 
permeable boundaries for the crew (not so underbounded that the crew operates only 
as individuals, but not so overbounded that they exclude information or assistance 
available outside their group per se) which can be useful later in the conduct of work 
on the line. They elaborate and expand the norms regarding safety, cooperation and 
communication. Under their leadership, new ways to share their authority emerge, and 
hence the total authority of the cockpit and cabin crews expands and becomes more 
effective. They create conditions which can lead to better crew performance by 
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expanding previously defined shell structures. These behaviors also tend to enlarge 
each crewmember’s concept of what the shell can be for an effective crew, and this 
improved image can be imported into the shells of subsequent crews of which they will 
be a member. 


3.8. Conclusion 


Prior to the first meeting of the crew, we find a collection of individuals, each with his or 
her own perception of the shells for crew behavior. That imported shell is only that—a 
shell which the captain can enhance or diminish. Captains can expand it or undermine 
it; they can affirm it or abdicate. But when the first meeting is over and the crew goes to 
work, they are some sort of a team. They may start work envisioning new and creative 
ways to improve team effectiveness, or they may be wondering what this crew is really 
going to be like. In one form or another, this new team now has its own shell, one 
shaped by the captain’s behavior regarding the tasks, by the boundary definitions the 
captain described, by the transmission of implicit and explicit norms, and by the 
authority dynamics demonstrated by the captain. If we assume that the company believes 
in CRM and provides sufficient shell support for crew work, then whether the captain 
enhances or impedes a crew’s ability to perform well is really up to him or her. 
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Introduction 


Communication is one of those boundless topics with many meanings and many uses 
because it is so fundamental to human endeavor. Whether written, verbal or non-verbal, 
face-to-face or remote, communication is an essential part of the social sciences: 
psychology, sociology, political science, sociolinguistics, etc. In addition to traditional 
academics, communication is pragmatic; that is, we communicate in order to acquire 
what we need and to accomplish goals. Thus, we are likely to think of communication in 
terms of success; you are understood or misunderstood, information is transmitted or it 
is not; you are persuaded or unmoved. Communication skills help to determine the 
success or failure in achieving goals and when one’s goals are attached to high stakes, 
communication effectiveness is essential. 

There is no doubt that operating in today’s airspace is a high-stakes profession since 
lives and costly assets are invested in every flight. As in other complex, human-technical 
systems, communication plays an important role in accomplishing goals, coordinating 
individuals and integrating tasks. In this chapter, we would like to underscore the 
importance of communication for safe and efficient flight operations and its role in 
achieving task goals and enabling Crew Resource Management (CRM). 

This chapter retains much of its structure from the original 1993 chapter as the basic 
concepts and history maintain their relevance. However, the contents have been updated 
to reflect how communication as a CRM skill has evolved over the last 15 years. The main 
differences lie in the last section of the chapter. CRM training and evaluation practices have 
matured a great deal since 1993 and the way communication is now trained and evaluated 
reflects comparable conceptual growth. No longer is it considered a generic soft skill that 
“you can recognize when you see it,” but is hard to pin down. Rather, training and 
evaluation—particularly in the simulator—have evolved, and communication indicators 
are tied to specific performance objectives within flight phases and under particular 
operational conditions. This has solidified our understanding of how good communication 
skills support CRM, effective crew performance and, ultimately, flight safety. 


4.1. Historical View of Communication and 
Flight Safety 


4.1.1. NTSB Accident Reports 


Probably the most dramatic and compelling demonstrations of the link between 
communication and flight safety come from the accident investigations carried out in 
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the USA by the National Transportation Safety Board (NTSB). Consider the example of 
Avianca flight #052, a Boeing 707B from Medellin, Columbia, to John E Kennedy 
International Airport (JFK), New York, which ran out of fuel over Long Island on 
January 25, 1990 (NTSB, 1991). Several critical failures in communication were 
evident; specifically, the crew failed to communicate to air traffic control (ATC) the 
information that they were desperately low on fuel and needed immediate clearance to 
land. Poor weather conditions led to the flight being held three times by ATC for a total 
of 1 hour and 17 minutes. Not until the third period of holding did the flight crew 
report that: 


1. the airplane could not hold longer than 5 minutes 
2. it was running out of fuel and 
3. it could not reach its alternate airport, Boston-Logan International. 


Following the execution of a missed approach to JFK, the crew experienced a loss of 
power to all four engines and crashed approximately 16 miles from the airport. 

The NTSB attributed probable cause of the accident to the failure of the flight crew 
to manage the airplane’s fuel load adequately and their failure to communicate an 
emergency fuel situation to ATC before fuel had been exhausted. Safety issues which 
included additional problematic communication links included the following: 


1. Pilot responsibilities and dispatch responsibilities regarding planning, fuel 
requirements, and flight following during international flights 


2. Pilot to controller communications regarding the terminology to be used to 
convey fuel status and the need for special handling 


3. ATC flow control procedures and responsibilities to accommodate aircraft with 
low fuel state and 


4. Flight crew coordination and English language proficiency of foreign crews 
(NTSB, 1991, p. v). 


In Figure 4.1 critical communication links are depicted by bi-directional arrows. 
Although the probable cause of the accident is attributed to Link #2, at least four sets of 
communication/information links were called into question. 

Effective communication among crewmembers has always been an essential 
component of the concept of crew coordination. The first NTSB mention of “flight 
deck resource management” was made in the report filed on the crash in 1978 in 
Portland, Oregon, of United Airlines flight #173 (NTSB, 1979). The probable cause 
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Figure 4.1 Critical information links in the Avianca flight #052 accident (NTSB, 1991) 


was determined to be failure of the captain to monitor aircraft fuel state, which resulted 
in total fuel exhaustion. Contributing causes included “the failure of the other two flight 
crewmembers either to fully comprehend the criticality of the fuel state or to 
successfully communicate their concern to the captain” (NTSB, 1979, p. 29). 

One of the results of that investigation was the FAA Air Carrier Operations Bulletin 
Number 8430.17 (Change 11), which gave instructions regarding resource management 
and interpersonal communications training for air carrier flight crews. This action was 
taken in response to one of the four recommendations made by the NTSB that focused 
on both participative management for captains and assertiveness training for other 
cockpit crewmembers. Since 1979, the NTSB has continued to consider the possible 
impact of crew resource management and crew communication in accident sequences. 

However, NTSB reports also acknowledge instances of exemplary CRM in their 
findings on the basis of communication data provided by the cockpit voice recorder 
(CVR). Possibly the most dramatic cases are United 811 (NTSB, 1990a) and United 232 
(NTSB, 1990b), in which flight crew interactions were “indicative of the value of 
cockpit resource management training which has been in existence at UAL for 
a decade” (NTSB, 1990b, p. 76). An analysis of the CVR communications (Predmore, 
1991) has identified specific communication patterns that may have contributed to the 
exemplary CRM (described in section 4.2.5). 


4.1.2. Incident Reports 


In contrast to accident reports, incidents reports are generated and collected in far greater 
numbers by a variety of organizations (e.g. airlines, unions and nationwide government 
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databases such as the NASA Aviation Safety Reporting System (ASRS)). These reports 
are usually voluntarily submitted and cannot be assumed to represent an unbiased 
perspective on all parts of the aviation system. Nevertheless, greater numbers of classes of 
events can indicate recurrent trouble spots. Furthermore, the US-wide sample of reports 
tells us whether these problems occur across the aviation system or whether they are 
specific to particular geographic regions, weather conditions, airports, airspace, etc. 
Because voice recordings are not available in incident reports, the role of 
communication is not directly observed. For example, an incident classified as 
a “workload management” problem may be, in part, brought about by a pilot’s 
ineffective communication style. Face-to-face communications within the flight deck or 
between flight crewmembers and ground support teams may not be so easily recognized 
or described as a communication problem. In short, level of description is left to the 
individual reporter who may or may not provide much detail in their narrative account. 
While communication problems cannot be analyzed at a word-by-word “transcript” 
level, incident data can accommodate higher level analyses. For instance, Billings & 
Cheaney (1981) analyzed transfer of information problems in the aviation system and 
found that over 70% of 28,000 reports submitted by pilots and air traffic controllers 
(during 1976-1981) fell in this category. Reports focused on pilot/controller 
interactions and controller to controller communications more often than within- 
cockpit communications. 


Close examination of ASRS reports led to the finding that information transfer 
problems...did not ordinarily result from an unavailability of information nor because the 
information was incorrect at its source...Instead, the most common findings showed that 
information was not transferred because (1) the person who had the information did not 
think it necessary to transfer it or (2) that the information was transferred, but inaccurately. 
(Billings & Cheaney, 1981, p. 2) 


In spite of the limitations imposed by voluntary reporting, incident reports are 
extremely informative with respect to identifying problem areas in many parts of the 
aviation system. Communication problems involve a variety of individual failures (e.g. 
distraction, failure to monitor, complacency) as well as system factors (e.g. radio 
frequency saturation, high workload) that interfere with successful information transfer 
(see Billings & Cheaney, 1981, p. 86). The identification of such behaviors and system 
factors not only informs the operational community but assists systems designers and 
researchers by pointing out areas of risk. For instance, how are communications affected 
in an automated environment or in conjunction with visual displays with aural alerts? 
How do information transfer problems surface under conditions of work overload, 
ambiguous data or failing equipment? 
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Based on reports to ASRS, it is concluded that information transfer problems are responsible 
for many potentially serious human errors in aviation operations. Voice communications, in 
particular, are a pervasive problem. Technological solutions exist for many problems related to 
information transfer. These solutions, however, may give rise to serious new problems unless 
they are implemented with an understanding of the capabilities and limitations of the 
humans who operate the aviation system. (Billings & Reynard, 1981, p. 13) 


4.1.3. Early Communication Research 

Early analyses of aircrew communication used three different data sources: 
1. Accident transcripts from the cockpit voice recorder (CVR) 
2. Real-time field observations and 
3. Full-mission simulations. 


Each of these sources provided different types of data; therefore they required 
different analysis methods. 


Accident cockpit voice recorder (CVR) transcripts 


Outside the realm of official investigations, one of the first systematic analyses of 
communications from CVR data was performed by Goguen et al. (1986). Hypotheses 
grew out of the recognition that assertiveness training may be needed for junior 
crewmembers, as recommended by the NTSB (NTSB, 1979). In order to study 
crewmember assertiveness, a classification scheme was devised to distinguish levels of 
mitigation (i.e. direct communication versus softened communication), and included 
speech types such as planning, explanation, and command and control. For example, 
a command stated in the imperative form is less mitigated than a suggestion which is 
usually spoken as a question. In order to consider whether captains were engaging 
crewmember participation, levels of mitigation were compared across positions, captain 
vs. first officer (FO), or second officer (SO). Results, based on eight transcripts and 
1,725 speech acts, included the following: 


1. Subordinate crewmembers were characterized by a more mitigated (softened) 
style of making requests 


2. Mitigated speech was associated with subsequent changes of topic and 
ungratified commands, indicating that mitigated communications were less 
successful and 
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3. Requests were less mitigated during conditions of recognized emergencies or 
problems. 


While mitigated requests appeared to be less effective at eliciting a response in general, 
this speech pattern usually occurred during less critical phases of flight. In contrast, 
requests were less mitigated (hence more effective) during times when there were 
recognized problems. In short, the use of mitigated speech did not seem to be a simple or 
general practice. Rather, mitigated communications seemed to be serving different 
purposes during different flight conditions. For example, the use of suggestions rather than 
commands during a pre-departure briefing may be a means of encouraging crewmember 
participation even though it may be an ineffective strategy during a critical phase. 


Field studies 


Field studies have the advantage of having uncompromised face validity; that is, there is 
no question that the behaviors observed are relevant to operations. While observations 
may be limited to what can be documented in real time and may represent mostly 
routine conditions, observations during actual operations can be an effective way to 
define problem areas and generate hypotheses. 

Field studies can go beyond mere observations. Costley et al. (1989) were among the 
first to develop an real time communication coding system for making systematic 
observations during flight. They investigated communication differences across aircraft 
types—B737-200, B737-300, and B757—which represented three levels of aircraft 
automation. The coding system included speech categories such as commanding, 
reacting, information processing, giving explanation, checking, summarizing, asides 
(jokes, quips) and questioning. From ten observation flights, two main communication 
differences were included: 


1. Lower communication rates in more automated aircraft (B737-200 versus 
B757) with no accompanying decrease in operational actions and 


2. Of the categories of speech affected, less questioning in more automated 
aircraft was the primary difference. 


While these data are suggestive of potential problem areas, the differences in communication 
were not linked to observed differences in performance. 

In general, if performance differences are not found to be linked to differences in 
communication patterns, we can assume they may be differences of lesser consequence. 
However, if (as discussed by Costley et al.) communication differences are linked to 
performance decrements under certain operational conditions, there could be serious 
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implications for communication training and interventions. We noted in 1993 that the 
results of this field work could be further tested by additional field work or in a flight 
simulator where some of the uncontrolled variables could be systematically manipulated 
or held constant. 


Simulation studies 


Analyses of accident transcripts and field research began to shed light on the role of 
communication in crew performance and flight safety. But high-fidelity full mission 
simulation research introduced a new compelling dimension to assessing crew 
performance. Because flight scenarios and conditions could be controlled and 
variables manipulated, flight crew performance could be measured with more 
statistical rigor. Unlike any previous source of communication data, the entire flight 
performance (including pre-pushback) could be videotaped. With this enhanced 
opportunity to investigate the relationship of communications to performance 
differences, full-mission simulation became a unique and powerful tool for 
communication researchers. 

The Ruffell Smith (1979) simulation at NASA Ames Research Center was 
a landmark study. In addition to demonstrating the yet untapped potential of 
high-fidelity, full mission simulators for research and training purposes, it confirmed 
what instructors, practitioners and accident investigators already knew. Technical skills 
alone were not enough to guarantee effective crew performance. More important, it 
proved that specific CRM behaviors, such as crew communication, and timely 
coordination could be clearly identified and characterized. 

Foushee and Manos (1981) expanded the Ruffell Smith study by analyzing the 
communication behaviors generated. The methodology included: 


1. a systematic “speech act” coding of verbatim transcribed speech 


2. an exploration of communication patterns that reliably related with 
differences in crew performance 


3. a test of specific effects of crew factors and 


4. control for operational conditions (e.g. normal versus abnormal operations) 
that could influence the use of particular speech patterns. 


The overall objective was to identify specific communication patterns associated with 
effective CRM outcomes so that training of best practices could be developed. 

While simulation methodology clearly offered the most research control and ability 
to investigate specific hypotheses, field studies and analyses from CVR data continued 


Communication and Crew Resource Management « Chapter 4 


to contribute to identifying issues and specific research questions. Regardless of the 
research method, it was useful to fit these research questions into a conceptual 
framework that described the relationships among the variables and outcomes of 
interest. 


4.1.4. The Communication Concept 


The relationship between communication patterns and practices with crew performance 
constitutes two parts of a three-part conceptual model of factors affecting group 
performance. This model, derived from McGrath’s (1984) theoretical framework of 
input, process and output variables, was altered to fit the aircrew work environment. See 
Chapter 1 for a discussion of the models depicted in Figures 1.2 and 1.3. 

To briefly review the model, input variables refer to a wide range of factors: 
attributes of individuals, characteristics of the group itself and factors related to the 
environment including specific task parameters that define the work environment. 
Individual-level input factors include any aspect of a group member that could 
conceivably affect that person’s ability to be an effective crewmember, such as flying 
skills, personality, motivation, physiological state and interpersonal skills. Group-level 
factors are aspects of the group as a whole; structure, size, collection of skills, etc. 
Clearly, flight crews are relatively standardized since there are many individual 
requirements (pilot certification standards) as well as requirements demanded by the 
aircraft type and task that should ensure a common level of competency. 
Environmental input factors are focused on characteristics of the operating work 
environment; aspects of the task, level of difficulty and stress involved, design factors 
such as flight deck configuration including displays and specialized equipment. 
External environmental factors such as weather and aircraft conditions also fall in this 
category. It is in this area that we find research examining the effects of automation 
on performance and crew communication (Wiener, 1989; Wiener et al., 1991). The 
importance of this area since 1993 has grown exponentially and will continue to do 
so in the future. In the current era of next generation initiatives, new technologies 
(data communications, electronic flight bag, advanced navigation aids) as well as 
procedural changes to match new ATC technologies and procedures make this 
“input” area very relevant. 

Outcome variables shown in Figure 1.2 refer to the individual, organizational, 
mission and crew performance outcomes such as safety and efficiency, but the most 
salient output concern is the relative success or failure of a group to achieve the mission 
and crew performance objectives. Performance errors and crew ratings have typically 
been the chief outcome measures in communication research. 
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Group process variables represent mediators between inputs and outcomes; they 
refer to the means by which crews achieve specific performance outcomes. As shown in 
Figure 1.3, these processes are affected by the various inputs that feed into it. 
Communication processes are of central importance to the group activities that rely on 
verbal exchanges and information transfer. In addition, communication is often the 
behavioral indicator of other CRM functions such as decision-making, 
problem-solving, resource and workload management. While a more traditional 
research paradigm focuses on the relationship between input and output (performance 
variables), there is growing recognition of the importance of group processes as 
intermediate predictors of group success (Foushee & Manos, 1981; Foushee et al., 1986, 
Kanki et al., 1989). Group process variables are behavioral sequences that describe 
the interactions of group members. They include communication patterns as well as 
other resource management strategies. 

Process variables have also been directly associated with performance as predictors of 
outcomes, independent of input variables. For example, Foushee et al. (1986) 
demonstrated that patterns of communication among air transport crews who had 
recently flown together were more clearly associated with higher levels of performance 
than the patterns of crews who were flying together for the first time. These have been 
further investigated and described in research by Kanki & Foushee (1989) and Kanki 
et al. (1989, 1991a) which indicates that crews that share similar communication patterns 
appear to perform better as a team. Thus, group process analyses have shown that 
communication patterns can be associated with performance differences. 

Communication analyses involve exploring relationships between group processes 
and both input and output variables. Although the overall direction of influence in the 
model (Chapter 1, Figure 1.2) flows from left to right (i.e. culminating in outcome), it 
must be noted that group processes are dynamic and change over time. The model also 
contains continual feedback loops because performance outcomes can and do reflect 
back onto ongoing group processes. How a crew prepares and communicates during 
pre-flight affects takeoff and departure phases just as planning and preparing during 
cruise can greatly affect the arrival and landing phases. Consequently, crews that plan 
ahead may never experience high workload, therefore they may never need to invoke 
workload management strategies. In contrast, crews that “get behind the aircraft” may 
have to redouble their efforts in order to “get everyone back in the loop.” An early 
outcome of a well-organized contingency plan may lead to a relatively “quiet” cockpit 
with respect to communication, while an early outcome of ill-timed preparation may 
lead to a flurry of communications needed to re-prioritize tasks, solve problems and 
manage workload. The final performance outcome, however, may be similar for both 
crews if the latter crew catches up. 
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In addition to the normal interdependence of actions that flow from one flight 
phase to the next, the dynamic nature of group process and communication arises 
from the changing state of the aircraft and conditions affecting flight. Even with highly 
routine forms of communication such as checklists and briefings, other aspects of the 
operational environment (aircraft malfunction, weather, traffic) may create the need 
for immediate changes in communication content and form. When it does not, there 
can be a sudden lack of leadership or information exchange. Kanki and Smith (2001) 
describe the nearly complete lack of communications from the Air Florida flight #90 
accident: 


When Air Florida Flight 90 (NTSB, 1982) crashed into Washington DC’s Fourteenth 
Street Bridge, it was in a full stall, yet both pilots, qualified in stall recovery procedures since 
primary flight training, failed to verbalize anything...that would have triggered an 
automatic stall recovery response, that is, applying full power. Instead, the inadequate 
communication that did occur...was more an acceptance of fate than a last-ditch effort to 
correct the problem. (p. 114) 


In contrast to the Air Florida crew, who either did not recognize a problem or who 
did not communicate an emergency response, is the case where a crewmember actively 
shuts down the communication. From a now-infamous ASRS report, a first officer 
attempted to assist an extremely “negative” captain by reminding him of the correct 
speed, heading and altitude during an approach into Chicago O’Hare. The captain’s 
response, “You just look out the damn window,” is a good example of a situation in 
which authoritarian control was excessive, inappropriate and unsafe (Foushee & Manos, 
1981, p. 70). 

In summary, the communication concept is one that describes a dynamic process in 
which communication is a primary means by which individuals develop and coordinate 
activities in order to achieve goals. Variations in communication patterns are useful 
indicators of effective crew solutions as well as crew problems. In any case, 
communications must be interpreted within a task, operational environment and 
interpersonal contexts which change dramatically over time, sometimes in expected, 
routine ways and sometimes in unexpected, rapidly evolving ways. 

As both a skill and a tool for achieving objectives, communication patterns and 
practices can be linked to crew performance outcomes and CRM. However, some 
differences can be considered simple individual styles or even cultural differences that do 
not affect crew performance at all. How do we decide what communication patterns are 
important to crew performance and which are not? The answer can be found in the key 
functions that communication serves. 
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4.2. Functions of Communication 


The philosopher John L. Austin in an entertaining book called How to Do Things with 
Words (1962) described how words can be “used” in much the same way bricks and 
boards are used to build things. Austin discussed the ability of language not only to “say” 
things but to “do” things as well. For instance, saying “I promise to do X” means you 
have actually “made a promise” (as long as you were sincere and understood what 
you were saying). Speech not only accompanies actions but is action itself. 

Because communication serves so many functions, it provides an effective index of 
crew performance. Just by listening to the communications during a flight, we get many 
indicators of whether tasks are being performed according to normal procedures or 
whether problems are occurring. When problems do occur, we can tell if they are 
handled in a timely way or if the crew is falling behind the curve. With respect to CRM, 
we outlined five functions in the 1993 chapter which are only slightly paraphrased 
below. Although these might have been categorized a little differently today, these five 
are still among the most significant ways that communication affects crew performance. 
In some cases, it is the actual communication content that is most important; in other 
cases, communication is a tool for achieving CRM objectives: 


1. Communication conveys information. 

2. Communication establishes interpersonal/team relationships. 

3. Communication establishes predictable behavior and expectations. 

4. Communication maintains attention to task and situational awareness. 
5. Communication is a management tool. 


Although each of these communication functions can be studied as a topic in its own 
right, in reality, most communications fulfill several functions at the same time. For 
example, if the captain makes it a point to bring flight and cabin crews together for a 
pre-departure briefing, his or her communications serve several functions 
simultaneously. They provide operational information relevant to the flight; they allow 
the captain to establish a leadership relationship with the rest of the crew, and, finally, 
they help to establish predictability, because the crewmembers now know something 
about the captain’s management style and expectations. Even if the captain said 
something as simple as “we’ll follow SOP,’ this assures crewmembers that company 
protocol will be expected and should be followed. 

In communication research, any one of these functions may be investigated directly, 
or, as in most accident investigations, all functions may be considered in one analysis. To 
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Table 4.1 Communication functions and some of their associated problems 


Communication functions Related problems 


Communication provides information Lack of information, incomplete or inaccurate 


information 





Communication establishes team relationships Interpersonal strain, ambiguity, lack of 


leadership or role clarity 


Communication establishes predictable behaviors Non-standard, unpredictable behavior 
(SOPs and best practices) 





Communication maintains attention to task and Lose of vigilance, monitoring, situational 
monitoring (situational awareness) awareness 





Communication is a management tool: resources, Lack of or misdirected management of task, 
time, workload time, resources, workload 





return to the Avianca flight #052 example, four of the five functions are illustrated in the 
safety issues raised: 


1. 


The communication link between pilots and dispatch points to a failure on 
the part of the pilots to manage or effectively utilize their resources from dispatch 


The communication link between pilots failed in terms of situation 
awareness and monitoring 


The communication link between pilots and ATC illustrates the simple 
failure to transfer information, namely, their “emergency” status and 


The communication exchanges exemplify a lack of predictable behavior 
patterns, since the language difference between pilots and controllers failed to 
provide the usual redundancy of information (via intonation and other para- 
linguistic cues) pertaining to emergency states. 


Since communications are typically multifunctional, any given flight may be 
analyzed on any or all of these dimensions. However, each of the five communication 
functions above is typically associated with a subset of potential problems that can 
result in crew performance decrement (see Table 4.1). The next five sections discuss 
each of the communication functions and their associated problems. There will be 
overlap across topics, but each function is critical to key CRM elements. 


4.2.1. Information Transfer 


The traditional view of communication highlights the information transfer function of 
language. In the cockpit, there are numerous information sources used in flight 
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including: crewmembers, ATC, company, manuals, maintenance log, checklists, etc. In 
addition, information is obtained from aircraft instruments, light indicators, aural alerts 
and from the outside world (e.g. weather, other aircraft and airport conditions). 
Information (often communicated in numerical form) is often safety critical and 
therefore required to be discussed and implemented in a pre-defined standardized 
manner (e.g. briefings and checklists). Standard operating procedures not only specify 
when information must be obtained and acted upon, but often include a verification 
component as well (e.g. cross-check, or readback). 

However, when operations are normal and repetitive, the information exchange can 
easily slip into rote recitation. Even when the actual words are “correct,” the timing, 
intonation and attention may influence whether the communication is actually 
successful. There are critical differences between simply repeating a statement versus 
verifying the statement versus questioning the statement. Take the following statement 
as an example: “This reads in the normal range.” If the statement was repeated by FO 
with a questioning intonation, it may be that FO did not hear the captain. If it was said 
with the same rising intonation by the FO but it was not a simple repetition, it may have 
been the case that the FO was questioning whether the reading was “normal.” If the FO 
made the statement after the captain read an instrument, it may have been a verification 
of the captain’s reading. In short, intonation and many other contextual features 
contribute to the intention and interpretation of a statement. 

Information transfer is crucial in problem-solving situations because the resolution 
of the problem relies on gathering and communicating pertinent information. The types 
of speech acts that become salient in problem-solving include statements that: 

(1) recognize problems, (2) state goals and subgoals, (3) plan and strategize, (4) gather 
information, (5) alert and predict and (6) explain (see Orasanu, 1990). Based on the 
communications from previous simulation studies (Foushee et al., 1986; Chidester et al., 
1990), these categories of problem-solving talk were compared across normal and 
abnormal phases of flight for high-performing crews versus low-performing crews. 

Highlighting the information function of communication, it is not simply gathering 
task-relevant information that ensures good performance, but also the patterning of 
these communications that is critical. In Orasanu’s (1990) study, better performing 
teams generated communication patterns showing problem-solving talk during 
low-workload phases and increased interchanges of planning and strategy formation. 
Poorer teams failed to engage in planning communication during low-workload 
periods. Thus, when workload became high, the information gathering increased but 
was not as effective. 

Each scenario and specific problem generates its own list of pertinent information 
and speech categories are tied directly to these points. For example, in the analysis of 
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expert decision-making strategies, Mosier (1991) developed an information transfer 
matrix containing “items evaluated as important to making the critical decisions of 
each flight segment, as well as the checklists and procedures associated with the 
abnormalities of that scenario...Information solicitation and transfer were coded on 
the matrices beginning with the onset of the abnormal situation” (p. 268). The 
general strategy was (1) to consider how correctly and completely the communication 
data from each flight crew filled out the matrix and (2) to assess whether performance 
and correctness of decision outcomes were related to this measure of situation 
assessment. 

Both studies are good examples of how communications serve an informational 
function critical to problem-solving and decision-making. Communication analyses 
help to delineate what information is critical, when it should be solicited, the best way in 
which it should be integrated, who the possessors of information are and whether 
specific patterns can be linked to more effective performance outcomes. 

Moving beyond the cockpit, every flight has numerous information transfer events 
between pilots and ATC that are still conducted voice-to-voice on specified radio 
frequencies. While most of these communications are routine, errors are made both in 
speaking and hearing, which requires the process to be augmented by a verification step 
(readback or hearback). As mentioned in the ASRS incident research, information 
transfer has been a problem area for many years. Even now, a recent ASRS Safety 
Publication focuses on continuing pilot/ATC communication problems involving 
(1) false anticipation of ATC calls based on expectations, (2) language problems and 
(3) call sign confusion (ASRS, 2009). 


4.2.2. Interpersonal/Team Relationships 


Communication serves a social function when it helps to form team relationships and 

creates a work atmosphere that affects how crews perform their duties. This is one reason 
why the pre-briefing of the captain to crew is so important. As discussed in Chapter 3, 
part of the leadership role is to establish the social climate in which crewmembers are 

encouraged and expected to provide and receive information. In Ginnett’s field study of 
team building (1987) many of the leadership attributes suggested the types of speech acts 
that would distinguish effective leaders. For example, effective leaders: 


m Explicitly affirm or elaborate on the rules, norms, task boundaries that constitute 
the normative model of the organizational task environment. 


m Establish clear authority dynamics, as well as their own technical, social and 
managerial competence during team creation prior to flight. 
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These findings help to clarify the types of communications that enable effective 
leadership as well as when these communications are likely to occur. 

Personality of crewmembers also seems like an obvious determinant of interpersonal 
relationships. In 1993, several lines of research had investigated the links between 
personality and performance (Helmreich & Wilhelm, 1989, Chidester, 1990, Chidester 
et al. 1990), and between personality and communication (Kanki et al., 1991b). 
Summarizing some of the findings from the Chidester et al. (1990) simulation study of 
“leader personality” there were found to be three personality groups of captains. | The 
first group consisted of positive, instrumental skill/expressive (IE+) captains who were 
highly motivated, goal-oriented achievers and who were also concerned with the 
interpersonal aspects of crew performance. The second group of negative instrumental 
(I—) captains were also high on goal achievement, but had little regard for interpersonal 
issues. The third group consisted of negative expressive (Ec—) captains who had lower 
motivation for achieving goals and toward enhancing interpersonal relationships with 
other crewmembers. Results showed that the lower performing crews were led by the 
Ec— captains whereas the higher performing crews were led by the IE+ and 
I— captains. It was somewhat surprising that I— captains (the “authoritarian” captain) 
would show no decrement in crew performance since they had the potential to shut 
down communication. On the other hand, these captains were probably quite 
clear about their role, unlike the Ec— captains who, in essence, abdicated their 
leadership role. 

Kanki et al. (1991b) investigated the links between the three captain personality types 
and communication patterns. This analysis involved 12 three-person crews grouped by 
captain personality type. In spite of a small number of crews, the findings suggested that 
the negative expressive (Ec—) captains initiated communication proportionately less 
than the other captains. Specifically, they provided fewer observations and questions 
while their FOs asked more questions possibly compensating for an information deficit. 
Positive instrumental (IE+) captains initiated speech marginally more than their crews 
but did not dominate in terms of overall speech ratios. 

The research on communication and personality was interesting with respect to 
leadership, or lack of it. But of most importance was the ability to characterize poorer 
performance or lack of leadership in terms of communication patterns since changing 
one’s personality or affecting airline hiring practices was not really feasible. However, 





' Personality type was determined by a battery of instruments including the Expanded 
Personal Attributes Questionnaire (Spence et al., 1979, the Work and Family 
Orientation Questionnaire (Spence & Helmreich, 1978) and the Revised Jenkins 
Activity Survey C. 
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one could develop good policies, procedures or practices involving requesting and 
providing information. 


4.2.3. Predictable Behavior 


Effective team performance in complex task environments requires team members to 
integrate their activities in an organized, timely fashion and to make their actions and 
intentions known to others. Some tasks are completed simultaneously (CA and FO pre- 
departure flows), some tasks in sequence (performing the steps in emergency 
procedures), but all need to be completed at the appropriate times. For example, 
entering descent phase requires a sequence of tasks performed by pilot flying and pilot 
monitoring that allows changes to aircraft configuration, altitude, speed and course to be 
accomplished in a timely but controlled manner. 

Coordination of tasks among crewmembers is facilitated by the fact that pilots share 
knowledge and skills to a great extent. Standard Operating Procedures (SOPs) extend 
the shared knowledge base by setting up expectations about who is doing what and 
when. To the extent that both pilots have the same cognitive or mental representation 
regarding the general state of the aircraft (i.e. location, course, altitude, flaps 
configuration, etc.), the simultaneous or sequential coordination of tasks can be smooth 
and predictable. Using SOPs frees busy crewmembers from having to spend valuable 
time searching for and validating routine information and is designed to allocate 
workload in an effective way. 

Communication is an important way to accomplish SOPs. In checklists, for 
example, communication specifies what tasks need to be done, who should do them, in 
what order, and when they should be completed (see Degani & Wiener, 1990). SOPs 
may also arise out of company policies that become conventionalized ways of 
performing tasks and/or communicating. For example, the task of operating radios 
during flight typically falls to the pilot monitoring. The pilot flying can then assume that 
all incoming and outgoing communications are being dealt with and that the pilot 
monitoring can relay the relevant information upon request. 

From a communication perspective, conventionalized patterns of information 
exchange serve a similar purpose, i.e. to create expectations about how and when 
information is available. When information is made available in a predictable way, more 
efficient understanding and utilization of that information is possible. For example, the 
use of standard statements like “positive rate, gear up” makes accurate information 
transfer easier. Even if only part of the phrase is heard in a noisy cockpit, pilots will know 
what is being communicated and can act accordingly. The notion of conventionalized 
communication patterns also refers to the standardized exchanges between pilot roles. 
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For example, captains may be expected to issue more commands than first officers, and 
first officers may be expected to use more acknowledgments of communications (e.g. 
commands) than do captains. Other less obvious patterns (best practices) may distinguish 
high-performance crews from less effective crews. 

For example, in a study of communication processes and crew familiarity, Foushee 
et al. (1986) conducted a full-mission simulation study and found that crews that flew the 
simulator immediately after completing a trip together (post-duty condition) performed 
better than crews that had the benefit of rest before the simulation but had not flown 
together (pre-duty condition).” Expanding on these findings, post-duty crews, in 
general, used more statements of intent to perform actions, more acknowledgments of 
others’ communications and a greater amount of communications overall. Interestingly, 
first officers in post-duty crews expressed more disagreements with captains than first 
officers in pre-duty crews. 

It has been suggested (Foushee et al., 1986; Kanki & Foushee, 1989) that the time 
spent flying together before the simulation increased the ability of crewmembers to 
anticipate each others’ actions and understand the style and content of their 
communication. Post-duty crews therefore could adopt a more informed or “familiar” 
style in which first officers might be more willing to initiate directives or question 
captains’ decisions. Furthermore, while there may have been stronger flow of 
“bottom-up” communication (i.e. from first officer to captain), the authority structure 
was not impaired (captains still issued more commands than first officers). 

Taking a different approach with these data, Kanki et al. (1989, 1991a) have shown 
that similarity of communication patterns may be a distinguishing feature of 
high-performance crews. This research has attempted to demonstrate that 
high-performance crews (regardless of whether or not they had flown together) share 
similar communication patterns, while lower-performing crews show dissimilar 
patterns. For example, consistent with the earlier findings (Foushee et al., 1986) in four 
of five best-performing crews, captains and first officers generated essentially the same 
proportions of speech types (commands, questions, acknowledgments, etc.). The five 
lower-performing crews used in these analyses showed no consistent pattern of speech 
types. These analyses suggest that high-performance crews share similar patterns of 
communication. Thus, regardless of amount of time spent flying together, 





* The study was originally investigating the effect of fatigue on aircrew performance. 
But when the “rested” crews (pre-duty condition) performed less well compared to 
the “fatigued” crews (post-duty), the results which were counterintuitive (fatigued 
crews flew better than rested crews), the data were re-analyzed focusing on the 
pre-duty vs. post-duty factor. 
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high-performance crews appear to have very quickly reached levels of efficient 
information transfer and crew coordination, perhaps because their communications 
followed a more predictable form.” 


4.2.4. Task Monitoring and Situation Awareness 


It should be obvious by now that many of the communication functions overlap. For 
instance, problem-solving and decision-making require correct situation assessment 
and the planned sequential acquisition of pertinent information. Furthermore, CRM 
principles that underlie good management and leadership skills are very much linked 
to achieving situational awareness and effective distribution of workload across 
crewmembers. However, safe operations depend on maintaining vigilance during 
normal and non-normal operations alike; from extremely low workload (i.e. “boring” 
conditions) to extremely high workload (i.e. managing multiple problems at the same 
time). From a communication standpoint, we are interested in what kinds of 
communication patterns contribute to maintaining attention to tasks, effective 
monitoring and situation awareness under any of these various conditions. 

NTSB investigations have singled out several instances of exemplary CRM 
behaviors in the face of extreme emergency. United flight #811 (NTSB, 1990a) and 
United flight #232 (NTSB, 1990b) were two such flights. In both cases the captain cited 
training in CRM as contributing significantly to the overall effectiveness of the crews. 
With these characterizations in hand, an analysis of the verbal behavior of each crew was 
undertaken to explore how catastrophic events impacted the dynamics of crew 
interaction, and how CRM principles contributed to successful crew performance 
under stressful, emergency conditions (Predmore, 1991). 

Similar to other studies, the verbatim transcripts from the cockpit voice recorder 
(CVR) were broken into units of analysis classified by speaker, target (of the 
communication), time of onset and speech type. Categories of speech acts included: 


1. command-advocacy 


2. incomplete-interrupted 





* It should be noted that several communication analyses have used a subset of the original 
data from Foushee et al. (1986). The first analysis (Kanki et al., 1989) included ten of the 
total 20 flight crews: the five highest and five lowest performances. The later analysis 
(Kanki et al., 1991a) added the eight middle-performing crews. Communications from 
two of the original 20 crews were not analyzable due to irregularities in the simulation 


flight. 
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3. reply-acknowledgment 
4. observation and 
5. inquiry. 


Larger units of speech called action decision sequences (ADSs, roughly representing 
topics or types of operational subtasks) were then delineated. These included: (1) flight 
control, (2) damage assessment, (3) problem solution, (4) landing, (5) emergency 
preparations and (6) social. 

Once all speech was categorized, the distribution of communication units was 
graphically presented on timelines. For example, a graph depicting speech act categories 
is shown in Chapter 1 (Figure 1.10). Timelines were constructed that depicted ADS units 
broken down by crewmembers (captain, first officer, flight engineer, check airman). 
These graphs show that the distribution of topics or attention shifts drastically over time. 
As one would expect, some topics (such as social) completely drop out because other 
ADSs, such as flight control and landing, are now assigned high priority. This form of 
analysis allows us to see where problem solution and damage assessment fall in the 
timeline and how they gained attention without loss of attention to other ADSs. While 
we already know something about effective communication patterns related to problem- 
solving (see prior discussion on information transfer), these patterns show how multiple 
tasks are distributed and monitored over time. As stated by Predmore (1991), “The 
interactions of the crew of United Flight 232 were marked by an efficient distribution of 
communications across multiple tasks and crewmembers, the maximum utilization of 
a fourth crewmember, the explicit prioritizing of task focus, and the active involvement 
of the Captain in all tasks throughout the scenario” (p. 355). Although we can appreciate 
the information transfer, team relationships, and crew and workload management 
functions of the communications, these analyses provide graphical descriptions of how 
effective task monitoring is achieved under the most demanding emergency conditions. 


4.2.5. Crew and Workload Management 


To many, the management function of communication is at the heart of CRM (e.g. 
management of resources, time, crew, workload). Consider the following characteristics 
of a hypothetical crew: 


1. The interpersonal atmosphere of the cockpit is conducive to a good working 
relationship among crewmembers. 


2. Standard procedures and crewmember expectations are known and reliable. 
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3. Information is available and accessible. 


4. Crewmembers are in the loop and ahead of the airplane (i.e. situationally 
aware). 


In short, we have a “CRM-ready”’ crew. All that is lacking is the actual 
implementation of the plans, problem solutions, decisions, etc. that constitutes “CRM 
in action.” To state it sequentially, crews “form, storm and norm” and are now ready to 
“perform” (see Ginnett, 1987; Tuckman, 1965). To enable performance a manager 
is needed who can lead, distribute tasks, oversee and monitor the whole process. Enter 
the captain who holds command authority and ultimate responsibility. 

But captain leadership is only half of the equation. The other side of leadership is 
followership. There is a strong implication that all crewmembers participate to some 
extent in the management function, since each is a player who contributes to the 
coordination of the crew as a team. Consider a multiple team environment such as the 
C-5 military transport, in which there are several levels of “teams.” The entire flight 
crew, led by the aircraft commander (AC), consists of pilots, at least two flight engineers 
(FE and scanner), and as many loadmasters as are needed to carry out the mission. 
However, the crew is also composed of three operational subteams, each of which has its 
own leader, namely the AC, the primary engineer and the primary loadmaster who 
oversee each work group. There are many periods when these subteams work 
autonomously, while at other times the team is called together into a single unit led by 
the AC. For instance, before takeoff and on approach, the AC makes a “crew report call 
to stations,’ which assemble the team as one. 

Even within a two-person cockpit, there are continual shifts between times when 
both pilots are working together and when each is working alone (e.g. one is flying the 
airplane and the other is communicating with ATC). Each of these team combinations 
consists of some predetermination of roles and tasks. One can even characterize the 
pilot/ATC relationship as a “team” and discuss the predetermined roles and tasks 
implicit in the meaning and significance of their communications. 

Whether the actions taking place are produced by a single team unit or by 
subteams working in parallel, they must fit into a single flow. It is to this end that 
communication takes a management function, coordinating the crew’s actions. We 
typically think of directive speech acts as commands and suggestions, but this does 
not imply that managers/leaders necessarily accomplish their tasks by dictating crew 
actions in an authoritarian way. In fact, recalling the results of Orasanu’s work 
(discussed earlier), directive speech during abnormal operations will not promote 
smooth crew coordination if the proper groundwork (e.g. planning and sharing of 
information) has not already been accomplished. The good manager knows when to 
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Table 4.2 Typology of management strategies. 


Aircraft Environment People* 


Planning Studying approach procedures Planning routes Prioritizing tasks 





Acting Following glideslope Implementing routes Assigning tasks 


Evaluating/ Assessing pitch of aircraft Reporting a navigational Evaluating crew 
monitoring fix reached experience with task 





“Highlighted cells in the People column represent the management of crew resources, task management and 
workload distribution. 
Source: (Adapted from Conley et al., 1990) 


take the control and when to let the crew do their jobs; when to direct and when to 
monitor. 

Other studies have looked directly at management functions. For example, Conley 
et al’s (1991) analysis of communication takes a task management perspective and all 
communications were coded into a 3 X 3 matrix that differentiated aspects of crew 
coordination during flight (i.e. planning, acting and evaluating/monitoring) by content 
domain (i.e. aircraft, environment and people). In the Conley et al. (1990) study, this 
matrix was used to classify the management strategies shown in Table 4.2. Task 
management from the CRM perspective is best represented by the three coordination 
techniques under the “people” content domain (highlighted in the table). Note that 
commands probably fall most often in the “action” cell of the matrix, while planning 
and evaluating cells may contain any variety of speech acts. 

Managing crew resources in the modern airspace system is a complex task which 
may involve coordinating teams within teams and tasks within tasks, both within the 
aircraft and with others in the system. The ability to manage communication must be 
cultivated, for it is through good communication skills that effective crews invite the 
participation and contributions of a diverse group of team members, direct and integrate 
a complex flow of tasks, and monitor a dynamic operation that may require changes at 
any moment. 


4.3. Issues and Advances in Communication 


At the time of the first edition of Cockpit Resource Management, communication research 
was a means to understanding and describing aspects of CRM behaviors. Since then the 
communication analysis and research has expanded its reach with respect to CRM 
training and evaluation and has been usefully applied in domains outside the cockpit and 
outside aviation. Thus, it is even more important to discuss the nature and limitations of 
communication research. 
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4.3.1. Issues in Communication Research 


A primary issue in the discussion of CRM research is the degree to which results 
generalize to the “real world.” A first limitation is imposed by research approach. For 
example, field research in which a study is conducted during operations has face validity 
that is hard to match by other methods. However, field studies are constrained in ways 
that reduce their scope. Because field studies often require spending many hours traveling 
with flight crews this may limit the number of crews that can be reasonably sampled. In 
addition, a researcher has limited control over the environmental or operational 
conditions and therefore cannot easily control factors of interest. On the other hand, 
laboratory studies often lack the operational realism to confidently generalize findings to 
the real world, even if the conditions are carefully controlled and the data are reliable. 
What we felt in 1993 and continue to believe today is that full-mission simulation offers 
an excellent compromise, providing enough control to develop operationally realistic 
scenarios with experimental conditions. Furthermore, the benefits of full-mission 
simulation for research are greatly magnified in the training and evaluation realm. 

The high-fidelity full-mission simulation paradigm introduced by Ruffell Smith 
(1979) not only represented a methodological breakthrough for CRM researchers but 
was a useful vehicle for generating many CRM research findings. However, it is 
important to keep in mind that even in the best full-mission simulation, choices have 
been made by researchers based on research focus. These choices result in the selection 
of particular conditions and design manipulations that best fit the research question. At 
the same time these choices limit the findings to a reduced set of real-world 
operations. For example, specific kinds of problems are built into scenarios to create 
opportunities to observe pilots’ decision-making and crew coordination skills. But 
every scenario is necessarily bound to the particular conditions and problems 
incorporated, thus excluding many other variations. Therefore care must be taken to 
qualify the conclusions made on the basis of a single full-mission study, or a single line 
operational simulation (LOS) scenario. 

Finally, problem focus is constrained by simulation limitations (e.g. realistic 
simulations involving cabin crew are difficult) and the research objective. For example, 
from a CRM perspective, we often concentrate on communications within the flight 
deck and do not look deeply at the pilot/ATC linkages even though they have been 
shown to be crucial in accident and incident reports. A study of pilot/controller 
collaboration (Morrow et al., 1991) found that procedural deviations were more likely 
to have been made by pilots when ATC economized their workload by composing 
longer messages. In addition, procedural deviations were also associated with 
non-routine transactions (e.g. clarifications, interruptions/repeats, corrections, etc.). 
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The results pointing to these potential pilot/ATC tradeoffs suggest that (1) studies of 
pilot communication and workload may be defining the problem space too narrowly 
and should consider that both pilot and controller performances are affected by their 
negotiations and (2) there may be important training implications for both pilots 
and controllers when their communications are directly assessed. This is not to imply 
that all studies must incorporate pilot/ATC processes, but care must be taken to 
qualify conclusions made on the basis of studies that omit certain aspects of flight 
operations. 


4.3.2. Advances in Communication and 
Investigation 


Widespread use of communication analysis in both aviation and space event 
investigation makes use of verbal recordings and transcripts. Investigators have always 
used the cockpit voice recorder (CVR) data to help understand the events and 
circumstances surrounding an accident, but with the awareness and acceptance of CRM 
concepts, investigators began to adopt systematic methods of analyzing CVR transcripts 
and focus on CRM behaviors as a part of the more general consideration of human 
performance. It is now common in many countries to see a systematic analysis of 
communication from the CVR and a consideration of CRM behaviors. Such analyses 
have also been applied to space operations (NTSB, 1993; Kanki, 1995). 

Communication analysis methodologies have incorporated some of the 
communication principles mentioned earlier in this chapter; for example, an 
appreciation of the many functions that communication serves. An obvious example is 
the use of communication data to support the establishment of predictable behavior; 
one’s use of communication protocols especially in radio communications and the use of 
standard operating procedures such as checklists and briefings. Another common 
example is the use of communication data to support the presence or absence of team 
relationships. In some cases, communications may indicate confusion about leadership 
or tension that shuts communications down. Similarly, communications are examined 
for their role in maintaining vigilance and situation awareness or for managing task 
priorities and workload. 


Pegasus launch procedures anomaly 


On February 9, 1993, in the final few minutes of countdown for the commercial launch 
of two satellites, confusion in the control room resulted in the continuation of the 
launch in spite of range safety’s call for abort. The communication timeline starting at 
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Table 4.3 Simplified timeline of communications for Pegasus launch procedure anomaly 
(NTSB, 1993) 


Time Communication 

T —3.28 discussion of an altitude deviation found in the data 
RSO calls for mandatory Abort; stops countdown clock 
TC tells NASA-1 of Abort, but recycle possible 
NASA-1 tells B52 “Abort” 











Abort discussion between TC, RCO and RS3: TC misinterprets nonverbal wave from 
RS3 to mean Negative on Abort 


T —.18 NASA-1 to B52 “Keep going” 

T —.08 TC "Go for launch” 

T —.06 NASA-1 to B52 "Go for launch” 
T —.04 B 52 "Go for launch” 

T —.02 RCO “Are you saying Abort” 
T.00 RCO “Abort Abort” 

T.00 B 52 "Pegasus Away” 


TC is the test conductor, RSO is range safety 
NASA-1 is the communicator to B52 launch pilot 
RCO is range control; RS3 is on range safety support staff 











59 seconds before launch is shown in Table 4.3. The timeline is simplified since the 
personnel locations and team roles are not indicated, but the communications show how 
the incident unfolded. 

Underlying the basic miscommunication were a number of contributing factors, 
many of which reflected CRM inadequacies such as: misuse of communication 
channel assignments resulting in lack of cross-team access to information; confusion 
about leadership authority; inconsistent use of communication protocols; lack of 
situation awareness; and ineffective problem-solving, decision-making and time 
management. 


Communication context 


In addition to serving many functions, communication occurs in a context. Information 
from these contexts contributes to how communication is conducted, what is 
communicated and how effectively communication is received. At least four contexts are 
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relevant when considering communication events: (1) the physical context, (2) the social 
and organizational context, (3) the task and operational context and (4) the speech and 
linguistic context. When interpreting what is communicated, these four contexts 
typically come into play; thus, they must also be considered when interpreting or 
evaluating crew communication. 

The physical context includes aspects of the work environment such as a noisy or 
quiet cockpit. It also includes whether communication takes place face-to-face or 
remotely. Face-to-face speech is often abbreviated because the communicators share 
the same situation and may be looking at the same thing. Speech may also be 
accompanied by pointing (to an indicator or display), gaze direction, etc. In an 
analysis of a transcript there may be no hope of understanding a statement like 
“Tt doesn’t look right” if you have no visual reference to what “it” is. 

The social and organizational context pertains to work role differences such as 
pilot-to-pilot speech compared to pilot-to-ATC communication. In each case, there is 
a work realm that greatly narrows what the communication means. A role difference 
within the cockpit is captain versus FO where the authority difference changes the 
meaning of statements that might otherwise seem identical. When the captain says 
“Would you like to pull out the checklist,” it is probably a command; when the FO says 
the same words, it is probably a question. In the analysis of an accident transcript, these 
roles and organizational contexts are often pre-established by regulations and company 
policies. 

The task and operational context is best described by the phases of flight 
and routine versus non-routine operations. Recalling the results of the early 
communication by Goguen et al. (1986), mitigated speech (which tended to 
be a less effective form of making requests) was not as frequently used during 
conditions of recognized emergencies or problems. In short, speech patterns 
were used differently during critical flight phases and may have served different 
functions for captains versus subordinate crewmembers. In accident transcripts 
there is not always an opportunity to compare normal phases of flight compared to 
non-normal phases, but what is critical is whether the crew recognizes their 
operational state. 

The speech and linguistic context consists of the grammatical and discourse rules of 
the language that would specify such patterns as “answers follow questions,” or the 
difference between completed speech versus speech fragments. It may also reflect 
language or cultural differences that could lead to misunderstandings of particular terms 
or phrases as in the Avianca accident. Aviation is considerably more standardized in its 
use of procedural speech than many other professions, but these regularities may fall 
short under unusual or emergency conditions. 
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4.3.3. Advances in Communication and CRM 
Training 


In the early days of CRM, communication was considered one of a number of CRM 
skills. In the curriculum suggested in the NASA/MAC CRM Workshop (Orlady & 
Foushee, 1987), communication was listed as one of seven major topic areas: 


1. communication 
2. situation awareness 
. problem-solving/decision-making/judgment 


. leadership/followership 


3 
4 
5. stress management 
6. critique and 

7 


. interpersonal skills. 


As a stand-alone skill, communication skills were described in a fairly long laundry 
list of examples such as “polite assertiveness and participation,” “active listening and 
feedback,” “legitimate avenue of dissent,” etc. (Orlady & Foushee, 1987, p. 199). Such 
examples were right on point, but training communication as a stand-alone skill was 
hard to pin down. As Kanki and Smith observed (2001): 


...communication often is relegated to a module in the Human Factors or CRM 
syllabus...When trainers isolate communications as a stand-alone “soft” CRM skill, they 
overlook the limitless potential of communication—the mechanism for achieving proficiency 
in technical, procedural and CRM skills. The implications of communication as a tool for 
achieving objectives permeates all aspects of instruction and evaluation, just as it does in 
actual flight operations. (pp. 110-111) 


Once communication was also considered as an enabling skill for technical, 
procedural and CRM objectives, it became much easier to specify communication 
indicators. In the FAA Advanced Qualification Program (AQP) the distinction between 
performance objectives and enabling objectives provided an appropriate and systematic 
way to link communication to the wide variety of objectives it can support. 

When communication is also considered in its operational context, the finer details 
of communicator roles (CA versus FO, pilot flying versus pilot monitoring), flight 
phases, and routine versus non-routine are incorporated. At times communication 
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behaviors are proceduralized (ATC communications, checklists, briefings), but often 
they are simply the tools for accomplishing a performance objective. Adapted from 
Kanki and Smith (2001), Table 4.4 provides examples of the main communication 

enabling skills used in routine operations and the functions that are added in non- 

routine operations. 

In Table 4.4, it is important to note that the objectives reflect a pilot-centric 
view. For example, while there is mention of ATC and company, these functions are 
not usually incorporated into line operational simulation (LOS) in a realistic way 
(Burki-Cohen et al., 2000). When Lee (2001) conducted a simulator study in which 
crews given realistic radio communications (RRC) (including simulated ATC, 


Table 4.4 Communication enabling skills for achieving objectives during routine versus 
non-routine operations 


Routine functions Additional non-routine functions 


Technical Flight Control: standard commands, Flight Control: time critical diagnosis, crew 
Objectives minor workload redistribution coordination 
Navigation: clarify and execute flight Navigation: complex planning with ATC and 
plan, contingency plans, programming company 
FMS Response to Hazard: enhanced, verbalized 
Systems Management: adjustments, situation awareness 
monitoring Systems Management: responding to uncer- 
tain requirements and changes 





Procedural Checklists: normal Checklists: non-normal/emergency 
Objectives Briefings: standard Briefings: standard and contingency 


ATC: standard Procedures: identification and performance 
of non-normal procedures 
ATC: problem resolution 





CRM Leadership: set the tone, vigilance, Leadership: set the tone, utilizing resources, 
Objectives team building setting priorities 
Monitoring: normal systems, route, Monitoring: normal and non-normal, 
traffic, flight efficiency, weather continual; dynamic event consequences, 
Task/Workload Management: minor time management 
adjustments, adherence to established Task/Workload Management: major 
routines and procedures adjustments, crew coordination 
Problem-Solving: time limited inquiry, 
assertiveness, advocacy of changes 
Decision-Making: time critical, risk sensitive, 
explicit online planning and executing 





Source: (Adapted from Kanki & Smith, 2001, p. 105) 
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dispatch and frequency chatter) were compared to non-realistic radio communication 
crews, the average time to execute a go-around more than doubled for the 

RRC crews. In short, realistic radio communications actually require much more 
time due to increased captain coordination with FO and FO with ATC. 


4.3.4. Advances in Communication and CRM 
Evaluation 


As an enabler for technical, procedural and CRM objectives, communications are often 
the prime indicator for an evaluator to assess whether specific objectives were achieved. 
In the LOS environment, major advances in event set methodology have improved the 
evaluation process. At the same time, it has allowed evaluators to use communications as 
indicators of technical, procedural and CRM objectives in a fairly straightforward way. 
Other chapters discuss LOS scenario development and instructor/evaluator tasks in 
detail, but it is only because LOS has incorporated a systematic method for designing 
a system of triggers and behavioral indicators that evaluations can be more easily and 
reliably performed. As long as we can clearly identify the proficiency objectives and their 
enablers, we can know what to look for and listen for. In many cases they will be 
communications. 

Event sets within flight phases provide specific operational contexts within which 
certain crew behaviors are expected to occur. Behaviors are even more finely 
distinguished by role (CA versus FO) or (pilot flying versus pilot monitoring). While 
many technical objectives and procedural objectives are required in every scenario 
because they are a part of normal operations, key behaviors can be evaluated by 
designing-in operational factors that require the target behaviors. Since the scenario is 
controlled, pilot options can be relatively well controlled and certain target behaviors 
should occur within an appropriate window of time. 

Because of the ability to design-in specific challenges, it is the optimal way to 
test, train and evaluate new procedures and technologies. With the appropriate 
changes in policies, procedures and possible new best practices, observing crew 
performance in the simulator can be an effective way to judge the adequacy of 
the new procedures and technologies as well as a way to determine what training 
will be needed. Through AQP and the design of effective LOS, monitoring 
crew performance can be achieved throughout fleet and company-wide 
implementations and address the reliability of instructor/evaluators at the 
same time. 
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4.4. Summary 


Communication has always been a part of CRM but it was originally considered 

a stand-alone skill. Accidents and incidents pointed to numerous examples of 
communication failures but the examples covered a wide range of communicators and 
situations. As result of a growing understanding of how CRM could affect crew 
performance, accident investigators and researchers began to systematically analyze 
communication that came from CVR transcripts, field studies and simulation studies. 
It became evident that: 


1. Communication is tool for achieving technical, procedural and CRM objectives. It may 
be proceduralized speech following explicit policies or it may be shaped by 
training as best practices. In LOS scenarios, it may be considered a behavioral 
indicator for specified performance objectives. 


2. Communications serve many functions. As a vehicle for: (1) information transfer, 
(2) establishing interpersonal or team relationships and (3) predictable behavior, 
(4) maintaining task monitoring and situation awareness and (5) ensuring 
effective crew and workload management, communication is both an enabler 
and indicator of these and other aspects of CRM. Any of these CRM objectives 
can be targeted with an appropriately designed LOS scenario. 


3. Communication occurs in a context and is interpreted in a context. These include: 
(1) the physical context, (2) the social and organizational context, (3) the task 
and operational context and (4) the speech and linguistic context. A part of 
training communication skills is knowing when communication is 
“appropriate” and “effective,” and much of this is determined by who the 
communicators are (social/organizational context), where the communication 
takes place (physical context), during what flight phase and under what 
operational conditions (task/operational context) and whether the 
communication is grammatically and culturally understood (speech/linguistic 
context). Using LOS event set methodology and invoking a well-designed 
scenario, all of these contexts are easily incorporated and serve as an information 
backdrop for evaluating communication effectiveness. 


Communication is a workhorse that serves CRM training and evaluation well. 
Within the CRM and AQP framework, and utilizing LOS events sets, observed 
communications are the behaviors indicators for many procedural and CRM objectives. 
Thus, communication as a practical skill has found a useful place in simulator training 
that can be scarcely matched elsewhere. 
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Looking to the future, however, there are always new benefits to be gained. The 
most compelling new area is actually an old area that has never been fully implemented, 
namely, the pilot/ATC interface. Because we anticipate a great number of new 
procedures related to data communications and navigation procedures, these changes 
need to be incorporated into the controller’s and pilot’s tasks. Ideally, some form of 
operational simulation with pilots and controllers could serve to test, train and evaluate 
the procedures as they are being developed rather than after they are implemented. 

Finally, flight deck technologies such as the Electronic Flight Bag (EFB) and other 
display changes will also modify the pilot’s task. As with other forms of automation 
technology, one fear is that pilots will keep their heads down while engrossed in their 
new information source, thus losing situational awareness. In addition, managing the 
EFB may create excessive workload during a time-critical flight phase if these tasks are 
not carefully integrated into an appropriately timed flow and distributed appropriately 
across pilot roles. Seamster and Kanki (2007) have demonstrated that the use of LOS can 
allow comparisons of the use of paper versus EFB and the assessment of workload and 
head-down behavior across the two conditions. As always, pilot communications are 
central to accomplishing workload management and ensuring situation awareness. In 
this and many other possible applications, the focus on communication and CRM 
within an AQP and LOS training framework provides an effective means for ensuring 
and enhancing crew performance into the next generation airspace system. 
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Introduction 


Flight crews make decisions all the time, from the captain’s acceptance of the aircraft and 
flight plan prior to departure to docking at the gate after landing. Unfortunately, the 
decisions that get the most attention are those that result in disasters, for example the 
decision to take off with snow and ice on the plane after the de-ice time had expired at 
Washington National Airport (NTSB, 1982), or the decision to take off without being 
sure the runway was clear of traffic in heavy fog at Tenerife, the Canary Islands (Dutch 
Aircraft Accident Inquiry Board, 1979). However, commercial aviation remains an 
incredibly safe mode of transportation, in good part due to the skills and judgment of its 
pilots. 

While an industry-wide analysis showed that over 70% of aviation accidents 
resulted from crew coordination or communication problems (as opposed to lack of 
individual technical skills, Lautman & Gallimore, 1987), the analysis of 37 aircraft 
accidents between 1978 and 1990 in which flight crew behavior contributed to the 
accident, the National Transportation Safety Board (NTSB, 1994) found that 25 
involved what the Board considered “tactical decision errors.” Line Operations 
Safety Audits (LOSA) involving observations of crew behavior during “normal” 
flights found that decision errors were among the least likely type of error to 
occur (~6%), but were more likely (along with proficiency errors) to become 
consequential (e.g. result in a hazardous aircraft state) than other types of errors 
(Klinect et al., 1999). Hence, maintaining safe flight operations depends on assuring 
effective crew decision-making, especially under threatening conditions (Helmreich 
et al., 2001). 

Because decision-making takes mental energy and because a large body of research 
suggests that people do not always make optimal decisions, aircraft designers, carriers and 
the FAA try to simplify crew decision-making by establishing standard procedures and 
checklists to cover anticipated failures or emergencies (Billings, 1991; Wiener, 1988), 
and through crew training and automated systems. However, poor decisions may occur 
even when problem situations are fairly straightforward because of conditions that 
increase risk, such as high workload, weather or heavy traffic. In other cases, simple 
problems cascade or interact, precluding “by the book” solutions. In still rarer cases, 
completely unforeseen catastrophic problems arise, like the loss of all hydraulic systems 
due to an engine explosion (NTSB, 1990). Given the impossibility of designing 
error-proof or fully automated systems that can cope with any and all emergencies, the 
last line of defense is the flight crew. Thus, the bottom line is: How can flight crews be 
trained and supported to make the best decisions possible, especially under challenging 
high-risk conditions? To address this question, we first must ask: How do crews typically 
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make decisions? What constitutes effective decision-making? What factors make 
decisions difficult and contribute to poor decisions? 

The short answer to these questions is the following: crew decision-making is not 
one thing. Crews make many different kinds of decisions, but all involve situation 
assessment and choice of a course of action that satisfies goals while managing risks. 
However, decisions differ in the degree to which they call upon different types of 
cognitive processes. A decision to abort a takeoff requires different decision processes 
from choosing an alternate airport for landing with a system failure or determining the 
cause of a master caution warning light. The nature of the processes involved in 
a decision depends on the structure of the decision task and the conditions surrounding 
it. How familiar is the problem? Is a response prescribed or must it be developed? How 
much time is available? Given the variety of decisions that are made routinely in the 
flight deck no single approach can be prescribed. No silver bullet exists to make crews 
better decision-makers. The long answer to the above questions is the subject of this 
chapter. 

Since the original version of this chapter appeared in 1993, the concept of 
decision-making in the cockpit has evolved, deepening its roots in the naturalistic 
decision-making (NDM) framework and growing toward the goal of threat and error 
management (TEM). Perhaps the biggest change in aviation decision-making (ADM) 
over the past 16 years is the concept of ADM as risk management. This shift emphasizes 
the importance of risk perception and risk assessment as essential components of 
effective decision-making. It also aligns error detection and correction with ADM, in 
keeping with the TEM framework (Helmreich, 2002). 

In this chapter the term ADM refers to decision-making by a flight crew, usually 
consisting of two members, but also includes solo pilots of varying skill levels and aircraft 
capabilities. 

This chapter is organized as follows: first, the processes by which flight crews make 
decisions are described; then we address factors that contribute to decision difficulty 
and poor decisions, followed by factors that provide crew resilience in the face of high- 
risk challenges. The final section explores strategies for improving crew decision- 
making. 


5.1. Aviation Decision-Making 


Aviation decision-making is viewed in this chapter as a form of “naturalistic 
decision-making” (NDM) (Klein et al., 1993). Naturalistic decision-making focuses 
on understanding how people with domain expertise use their knowledge to make 
decisions, typically in safety-critical environments (Cannon-Bowers et al., 1996; 
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Zsambok & Klein, 1997). Certain features characterize naturalistic decision-making 
and distinguish it from classical analytical decision-making (Lipshitz et al., 2001). 
According to Lipshitz et al., its essential characteristics are: (1) choice 
(conceptualizing decision-making as choosing among concurrently available 
alternatives), (2) input-output orientation (focusing on predicting which alternative 
will, or should be, chosen given a decision-maker’s preferences), (3) comprehensiveness 
(conceptualizing decision-making as a deliberate and analytic process that requires 
a relatively thorough information search, particularly for optimal performance), and 
(4) formalism (the development of abstract, context-free models amenable to 
quantitative testing). 

In contrast, in the NDM approach emphasis is placed on situation assessment prior to 
choosing a course of action; a process orientation replaces the input—output orientation; 
satisficing (i.e. achieving a “good enough” rather than optimal solution) replaces 
comprehensiveness and optimality; and context-sensitive informal models replace 
formalism (Lipshitz et al., 2001). These shifts stem from the fact that human information 
processing limitations! preclude exhaustive information search and simultaneous 
comparison of multiple options. Moreover, in many high-risk consequential 
environments, time for making a decision is limited, information is incomplete, 
conditions change dynamically, and goals shift, rendering analytic decision-making 
impractical, if not impossible. 

In NDM,, decisions are integral to a task and are made in order to achieve operational 
goals, such as transporting passengers to their destinations while managing any threats to 
safety. The decision-maker’s knowledge, often acquired through many years of training 
and experience, plays a key role in the decision process. Knowledge is the basis for 
recognizing situations that require decisions to be made, assessing the type and degree of 
threat present, determining what information is relevant to the decision, and deciding 
on an appropriate course of action. Also, team members, when present, expand the 
cognitive resources and help to overcome potential limitations of a single decision- 
maker. 


5.1.1. Theoretical Foundations 


Several naturalistic decision models contributed to the ADM model described in this 
chapter. Four were especially influential: Klein’s (1993a) Recognition-Primed Decision 
(RPD) model, Hammond’s Cognitive Continuum theory (Hammond et al., 1987), 
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Rasmussen’s (1993) Levels of Cognitive Control model, and Cohen’s (Cohen et al., 
1996) Recognition/Metacognition model. Each model contributed in the following 
ways to the aviation decision model. 


Klein—Recognition Primed Decision-Making 


Klein’s notion of how domain-specific knowledge guides recognition of situations and 
retrieval of appropriate responses is the foundation for many aviation decisions, 
especially those made under time pressure. Schema-based knowledge provides the link 
between recognizable situation patterns and actions that have worked in the past under 
similar conditions; this knowledge also is the basis for mentally simulating outcomes of 
the actions. 


Hammond—Cognitive Continuum Theory 


This theory asserts that decisions vary from intuitive to analytic depending on the nature 
of the situations in which they are made. In intuitive decisions, people rely on pattern 
matching; in analytical decisions, they use more thorough evaluation processes. 
Accompanying this continuum is a task continuum that reflects the nature of the cues 
available to the decision-maker. The amount and type of data determine where 

a problem sits on the continuum. According to Hammond, good decisions depend on 
correspondence between the decision strategy and the task: an analytical strategy applied 
to non-engineered cue patterns is inefficient, whereas an intuitive strategy applied to 
numeric data may be suboptimal. 


Rasmussen—Levels of Cognitive Control 


Rasmussen’s (1985, 1993) Cognitive Control model distinguishes between skill-based, 
rule-based and knowledge-based behaviors. Skill-based behaviors are highly practiced 
and exercised automatically in response to changes in cue input. Rule-based behaviors 
involve conscious action, but typically are codified in standard procedures. Knowledge- 
based behaviors involve deliberate thinking that is needed when skill- and rule-based 

guidance is lacking. Analytic or choice decisions and creative problem-solving fall into 
this category. 


Cohen—Recognition/Metacognition Model 


Some form of control process is needed to manage the shifts along a cognitive 
continuum, levels of cognitive control and types of recognitional processes. Cohen et al. 
(1996) formalized a critical thinking model that combined recognitional and 
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Figure 5.1 Schematic of aviation decision process model in response to an environ- 
mental threat. Decision processes are shown as a function of environmental condi- 
tions pertaining to cue diagnosticity and familiarity, time pressure, and risk, as well 
as response affordances: availability and familiarity of response options. 


Threat Event ==> CUE Pattern 
What's the problem? 
Situation Assessment How much time to resolve? 
How severe are potential consequences? 


Time Available Gather more 
Risk High or Low information 
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; Create Novel 
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metacognitive processes. The recognition component involves a quick test to assess 
situation familiarity and problems with the retrieved solution, time available to make 
a decision and the stakes if an error is made. If no problems are found, then the RPD- 
elicited response is adopted. But if stakes are high, time is available and the problem is 
atypical, then the metacognitive component is invoked. It provides a basis for critiquing 
and correcting the problem definition and solution until the decision-maker is confident 
of the best interpretation and solution. Cohen’s quick test for problem familiarity, time 
and stakes maps closely to the situation assessment processes in the ADM model depicted 
in Figure 5.1. 
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5.1.2. The Role of Expertise 


Expertise contributes to cockpit decision-making in three ways. First, expert knowledge 
facilitates rapid and accurate perception of cues and interpretation of problems 
(Cannon-Bowers et al., 1990). Second, this knowledge includes stored condition—action 
patterns, the basis for recognition-primed decisions (Klein, 1989, 1993). Third, expert 
knowledge provides a basis for risk assessment and for estimating the likelihood of 
occurrence of various kinds of problems. 

Knowledge is not a shield against errors. Expertise is the foundation for heuristics, 
which sometimes result in poor judgments. Deep knowledge is responsible for efficient 
functioning most of the time, but occasionally leads one astray. 

Expert knowledge confers an advantage primarily for problems that are meaningful 
within the expert’s domain (Klein, 1998). Chess masters show remarkable memory for 
the location of chess pieces that represent positions during play, a basis for strategic 
moves (Chase & Simon, 1973). But if those same pieces are placed randomly on the 
chessboard, the masters’ recall is no better than the novice’s. 


5.1.3. Aviation Decision Process Model 


The aviation decision-making model described in Orasanu (1993) still holds, with some 
minor adjustments as shown in Figure 5.1. Using NDM research methods, Orasanu and 
Fischer (1997) analyzed aviation incident reports (from the NASA Aviation Safety 
Reporting System, ASRS), National Transportation Safety Board (NTSB) accident 
reports and observations of flight crews in complex flight simulations to develop an 
aviation decision process model. The model involves two major components: situation 
assessment (SA) and choosing a course of action (CoA). 


Situation Assessment 


Aviation decisions typically are prompted by off-nominal or changed conditions that 
require adjustment of the planned course of action. Situation assessment involves 
defining the problem, assessing the level of risk associated with it and determining the 
amount of time available for solving it. Available time appears to be a major determinant 
of subsequent strategies. If the situation is not understood, diagnostic actions may be 
taken, but only if sufficient time is available. External time pressures may be mitigated by 
crews to support information search and problem solution (Orasanu & Strauch, 1994), 
e.g. crews may buy time through holding, or reprioritize tasks to reduce workload 
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(cf. Wickens & Raby, 1991). If risk is high and time is limited, action may be taken 
without thorough understanding of the problem. 


Risk Assessment 


Perhaps the most significant elaboration of the ADM model since 1993 is increased 
attention to the risk assessment component of situation assessment. In order to manage 
risk, threats must be perceived and accurately assessed. Risk includes two components: 
the likelihood of a threat and the severity of its potential consequences. The two vary 
independently, although in aviation, events with the most severe consequences typically 
occur quite infrequently. These include engine failures, fires and other emergencies. 
Other threats are more common, such as traffic, weather and schedule delays, but 
strategies for their management are more accessible. 

While relatively little research has been conducted on risk perception in aviation (but 
see O’Hare and Smitheram, 1995), Fischer et al. (2003) conducted a study that clearly 
demonstrated the role of risk perception in decision-making. Using a think-aloud 
protocol with dynamically evolving ambiguous flight situations, the investigators 
analyzed the information pilots requested, their evaluation of conditions in terms of risk, 
sensitivity to ambiguity, time available and goal conflicts, and ultimate decisions and risk 
management strategies. Findings showed that decisions—either accepting or avoiding 
risks—depended on how the pilots perceived the ambiguous conditions. If they felt 
conditions were not too severe, they were willing to take a risk, but if the risk level 
passed a threshold, they avoided the risk. 


Choosing a Course of Action 


After the problem is defined and the conditions are assessed, a course of action is chosen 
based on the structure of the options present in the situation. Building on Rasmussen 
(1985), Orasanu and Fischer (1997) specified three types of response structures: rule-based, 
choice and creative. All involve application of knowledge but vary in the degree to which 
the response is determined by the situation. What constitutes an appropriate course of 
action depends on the affordances of the situation. Sometimes a single response is 
prescribed in company manuals or procedures. At other times, multiple options exist from 
which one must be selected. On some rare occasions, no option is readily available and the 
crew must invent a course of action. In order to deal appropriately with the situation, the 
decision-maker must be aware of what constitutes a situationally appropriate process. 
While various types of decisions can be distinguished for analytical purposes, in 
practice any given flight situation may require the use of several different decision 
strategies. Making one decision or taking the prescribed action may present a new set of 
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conditions requiring a different type of decision. To an observer, these may appear as 
a smooth flow of action, although decisions are hidden behind the actions. 


5.1.4. Situational Constraints and Affordances in 
Choosing a Course of Action 


Rule-based or RPD decisions 


Single-response situations correspond to Klein’s (1989, 1993a) recognition-primed 
decisions and Rasmussen’s (1985, 1993) rule-based actions. Single option cases are the 
simplest decisions because they require the least cognitive work. 

In many high-risk and time pressured situations, such as smoke in the cockpit, an 
engine stall, or rapid decompression, an action is prescribed in response to specific 
situation cues. These and other abnormal situations are deemed to be sufficiently 
consequential that procedures are specified to reduce the crew’s need to invent solutions 
for the problem. The primary decision is whether any circumstances suggest that the 
pre-defined response should not be implemented. 

Sometimes an action may be planned or in process and a stimulus triggers a decision 
to terminate the action, a go/no go decision. Stimulus conditions that elicit this response 
may be quite diverse. For example, a rejected takeoff may be triggered by an explosive 
engine failure, cargo door light, runway traffic, compressor stall, or engine overheat 
lights (also see Chamberlin, 1991). Likewise, a missed approach—a decision to 
terminate a landing—may be triggered by inability to see the runway at decision height, 
by air or ground traffic, autopilot disengagement, or an unstable approach. 

All rule-based decisions involve risk assessment, particularly when ground speed or 
altitude is near a decision threshold. Certain conditions, like a wet runway or system 
malfunction that result in poor braking, will complicate the decision. 

Decision-making in aviation differs from many other high-consequence domains 
such as medicine or fire-fighting in that many rule-based aviatio decisions are codified in 
FAA regulations or company operations manuals. In other domains the basis for 
condition—action pairings is the decision-maker’s experience, which builds deep domain 
knowledge. Klein (1989, 1998) found that experienced decision-makers recognize a cue 
configuration as signaling a particular type of problem and then generate an appropriate 
response based on previous experience with similar problems. Once a response option 
has been retrieved, it is evaluated by mentally simulating its consequences to determine 
whether the response will satisfy the decision-maker’s goals. If so, the action is accepted. 
If not, another option is generated and evaluated, or the situation is reassessed. 
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Multiple-option decisions 


Some flight decisions involve choice from among alternatives present in the situation. 
These choices map most closely onto our everyday notion of decision-making. For 
example, a crew may need to select an alternate landing site in response to an on-board 
medical emergency in bad weather. Landing alternates are prescribed in the flight plan if 
weather conditions at the destination deteriorate, and procedure manuals provide 
guidance on how to deal with medical emergencies. However, weather conditions may 
be deteriorating at the nearest airport that offers appropriate medical facilities, and 
precious time may be required to reach a different airport. In this case, the crew needs to 
weigh the risks of trying to land in borderline weather conditions versus the possible 
danger to the passenger of flying to a more distant airport. 

Strategies used by crews to select from among alternatives vary, but observations to 
date (Klein, 1993a; Orasanu, 1993) suggest that they do not correspond to a full 
analytical procedure. A full analysis would involve evaluation of each possible option in 
terms of every variable relevant to the decision (e.g. weather, fuel consumption, runway 
length), and a mathematical formula would be used to combine all the information to 
yield the optimal choice. In fact, crews appear to make decisions in the most economical 
way, taking short-cuts in this process. They work toward a suitable—but not necessarily 
the best—decision in the shortest time, investing the least possible cognitive work. 

Options often are eliminated on the basis of one feature, such as weather, and are out 
of the running thereafter, unless no suitable alternate can be found and the process must 
be reopened. This is essentially an elimination by aspects strategy (Tversky, 1972). 
However, if a few candidates are available, one is chosen to match the constraints of the 
circumstances and the crew’s goals and perceived risks. Usually, the most safety-critical 
constraint prevails. However, organizational policy also plays an important role. The 
crew may choose an alternate that has a company maintenance facility or where 
replacement planes will be available for passengers to continue their flight. 


Ill-Defined problems 


Two other types of decisions hardly look like decisions at all. They consist of ill-defined 
problems that may or may not be clarified in the process of situation assessment. 
Ambiguous cues may make it impossible to define the problem that needs fixing. Two 
strategies may be used to cope with this type of situation: manage the situation as though 
it is an emergency without clearly defining the problem (procedural management), or 
diagnose and define the problem, and then generate a novel solution (creative problem- 
solving) because no prescribed procedures exist for dealing with it. 
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Procedural management. Certain cues are ominous but leave the crew without 
a clear idea of the underlying problem. Various noises, thumps, vibrations, 
rumblings, pressure changes, or control problems indicate that something has 
happened, but not necessarily what. The cues signal potentially dangerous conditions 
that trigger emergency responses, regardless of the source of the problem. Smoke, loss 
of pressure, an acrid smell, an explosion, or loss of control all signal “Land now.” Little 
time is devoted to determining the source of the cues. All energies are devoted to 
finding an appropriate airport, running necessary checklists, getting landing clearance, 
declaring an emergency, dumping fuel and landing. These problems are essentially 
treated as RPD situations, with the condition broadly labeled as “emergency landing.” 
A recent example of a procedural management decision was the landing on the Hudson 
River by a USAir A-320 aircraft after both engines were lost due to bird strikes on 
takeoff from LaGuardia Airport in New York (NTSB, 2009). Captain Sullenberger 
initially planned to land at Teterboro Airport in New Jersey after the dual engine loss, 
but realized they had insufficient altitude to travel the 6 miles; instead he opted to make 
a river landing. 

The cognitive work done for this class of decision is primarily risk assessment. 
Responses are clearly prescribed and highly procedural—once the situation is defined as 
an emergency. If the risk is judged to be high, then emergency procedures are 
undertaken. If the risk is not immediately defined as an emergency, then additional 
energy may be devoted to situation diagnosis. 

Diagnosis of the problem underlying ambiguous cues can serve two purposes. It can 
clarify what the problem is so that an appropriate action can be taken, or it can provide 
information that may be useful for fixing the problem. When workload is relatively low and 
time is available, the crew may try to diagnose and fix the problem. But even if diagnosis 
does not lead to fixing the malfunction, it can turn the problem into one with a better- 
defined response (essentially a recognition-primed decision). Defining the problem clearly 
may lead to a more specific response than simply treating it as an emergency. 


Creative problem-solving. Perhaps the most difficult types of decisions are those 
requiring creative problem-solving. In addition to diagnosing the situation, a solution 
must be invented that will satisfy the goal. These cases tend to be low-frequency events; 
neither aircraft designers nor operations personnel imagined such a situation would arise, 
so no procedures were designed to cope with them. 

Diagnosis is critical and typically involves causal reasoning, which is reasoning 
backward from effects to cause, as well as hypothesis generation and testing. The range of 
tests that can be performed will vary. For example, in response to a power loss indication 
for one engine, the crew can manipulate the throttle to see its effect. If they find no 
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effect, they may shut down the engine since it is not working. They may check to see if 
fuel is flowing to the engine. Tests often are embedded in checklists. 

Even if the nature of the problem has been determined, no ready solutions are 
prescribed for these problems. Perhaps the most celebrated case of creative problem 
solving was United Airlines flight 232 (NTSB, 1990) in which the DC10 lost all 
hydraulic systems due to an explosion in the number two engine. The captain invested 
considerable energy on situation assessment, determining what capability he had left 
after the hydraulic failure (Predmore, 1991). The two outboard engines were still 
running, but no flight controls were operative. Knowing that the only control he had 
was engine thrust, he and his crew determined that they could use asymmetrical engine 
thrust to turn the plane and power level to control the altitude. 

While the case of UAL 232 is extreme, ASRS reports indicate that crews do, in fact, 
encounter novel situations that are not covered by the Federal Aviation Regulations 
(FARs), Minimum Equipment Lists (MEL), or checklists. For example, the captain of 
a large transport on a cross-country flight reported a low level of oxygen in the crew 
emergency tanks while at FL310. No guidance concerning how to proceed was available 
in company manuals. The cause of oxygen depletion could not be determined in flight, 
nor could the problem be fixed. Regulations require emergency oxygen in case of rapid 
decompression, so rather than land immediately, the crew came up with a creative 
solution. They descended to FL250 and borrowed the flight attendants’ walk-around 
oxygen bottles. (Different O2 requirements are specified for flight attendants above and 
below FL250.) This solution allowed them to continue to their destination rather than 
to divert or to descend to 10,000 feet, which would have eliminated the need for the Oo. 
However, the latter option would have meant the flight would not have had sufficient 
fuel to reach its destination because of rerouting around bad weather. 

This example is interesting because it illustrates consideration of multiple options, 
creation of a novel solution, sensitivity to constraints and explicit risk assessment. In 
creating his solution, the captain was aware that he would not be able to communicate with 
ATC inan emergency ifhe were using the walk-around O3 bottle, as it had no microphone. 
But he judged the likelihood of a rapid decompression to be sufficiently low that he chose 
this option. Another constraint was fuel; the captain wanted to conserve fuel because of the 
possibility of a missed approach or diversion due to poor destination weather. An early 
decision to divert would have been the most conservative decision, but it would not have 
met the goal of getting the passengers to their destination in a timely manner. 

This above effort to classify decisions in terms of situational demands and affordances is 
a first step toward understanding what makes certain kinds of decisions difficult, the 
cognitive effort they require and possible weak links. The various types of decisions fall on 
a continuum ranging from simple to complex, which require little cognitive work to 
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considerable effort. One reason for laying out these differences is to create an 
appreciation for the fact that no single decision method will work for all types of situations. 


5.2. What Factors Make Decisions Difficult? 


Before examining factors that make decisions difficult and contribute to errors, the 
concept of “decision error” within an NDM framework must be considered. 


5.2.1. Decision Errors: Outcome vs. Process 


Defining decision errors in naturalistic contexts is fraught with difficulties. First, errors 
typically are defined as deviations from a criterion of accuracy. However, defining the 
“best” decision in a natural work environment may be impossible. Second, a loose 
coupling of decision processes and event outcomes works against using outcomes as 
reliable indicators of decision quality. Redundancies in the system can “save” a poor 
decision from serious consequences. Conversely, even the best decision may be 
overwhelmed by events over which the decision-maker has no control, resulting in an 
undesirable outcome. A third problem is hindsight bias. Fischhoff (1975), Hawkins and 
Hastie (1990) and others point out a tendency to define errors by their consequences. 
But in natural contexts the analyst does not know how often exactly the same decision 
process was used or the same decision was made in the face of similar situations with no 
negative consequences. Were those prior decisions also “errors”? 

Unsatisfying though it may be, the following definition is adopted here: decision 
errors are “deviations from some standard decision process that increase the likelihood 
of bad outcomes” (Lipshitz, 1997, p. 152). Although outcome alone may not be 
a good indicator of decision quality, the decision-maker’s goal or intended outcome 
remains important. In naturalistic work contexts, decisions contribute to performance 
goals. Decisions do not stand alone as events to be judged independent of the broader 
task. 


5.2.2. How Can Decision Processes Go Wrong? 


Decision errors may arise within the two major components of the aviation decision 
model: (a) pilots may develop an incorrect interpretation of the situation, which leads to 
an inappropriate decision, or (b) they may establish an accurate picture of the situation, 
but choose an inappropriate course of action. In addition, they may not appropriately 
assess the risks inherent in the situation. 
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Faulty situation assessment 


Situation assessment errors can be of several types: cues may be misinterpreted, 
misdiagnosed, or ignored, resulting in an incorrect picture of the problem (Endsley, 
1995); risk levels may be misassessed (Johnston, 1996; Orasanu et al., 2004); or the 
amount of available time may be misjudged (Keinan, 1988; Maule, 1997; Orasanu & 
Strauch, 1994). Problems may arise when conditions change and pilots do not update 
their situation models (Woods & Sarter, 1998). 

For example, one accident that can be traced to an incorrect assessment of the 
situation was the decision by the crew of a B-737 to shut down an engine; unfortunately, 
the wrong one: 


The crew sensed a strong vibration while in cruise flight at 28,000 ft. A burning smell and 
fumes were present in the passenger cabin, which led the crew to think there was a problem in 
the right engine (because of the connection between the cabin air conditioning and the right 
engine). The captain throttled back the right engine and the vibration stopped. However, this 
was coincidental. In fact, the left engine had thrown a turbine blade and gone into 

a compressor stall. The captain ordered the right engine shut down and began to return to the 
airport. He again questioned which engine had the problem, but communication with air 

traffic control and the need to reprogram the flight management computer took precedence, 

and they never did verify the location of the problem. The faulty engine failed completely as 
they neared the airport, and they crashed with neither engine running. (AAIB, 1990) 


The problem was incorrectly defined because the cues (vibration and burning 
smell) supported the interpretation of a right engine problem. The crew did not verify 
this interpretation before taking an action that was irrevocable at that point in the 


flight. 


Faulty selection of action 


Errors in choosing a course of action may also be of several types. In rule-based 
decisions, the appropriate response may not be retrieved from memory, either because it 
was not known or because some contextual factor mitigated against it. Conversely, an 
inappropriate rule may be applied, especially a frequently used one. In choice decisions, 
options may not be considered. Constraints that determine the adequacy of various 
options may not be used in evaluating them. 

Creative decisions may be the most difficult because they involve the least support 
from the environment; candidate solutions must be invented to fit the goals and existing 
conditions. Any solution that meets one’s goal may be considered a success. As Klein 
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(1993b) noted, decision-makers may not project the consequences and uncertainties 
associated with candidate actions, resulting in a poor choice of action. 

An accident in which an inappropriate course of action was chosen in the face of 
fairly complete information about the nature of a problem occurred near 
Pinckneyville, IL. 


About two minutes into a night flight in instrument conditions, a Hawker-Siddley commuter 
aircraft lost its left generator. In error, the first officer isolated the right generator and then was 
unable to restart it. This meant total loss of ability to generate electrical power, which was 
needed to run all cockpit instruments. Under the best of circumstances batteries might be 
expected to last for 30 minutes. The captain decided to continue to the destination airport 
45 minutes away, rather than diverting. Continued use of non-essential electrical equipment 
shortened the battery life. A complete electrical failure and subsequent loss of flight 
instruments critical for IFR flight led the plane to crash. (NTSB, 1985) 


The crew’s decision to continue as planned despite the mechanical failure, rather 
than to land as soon as possible, was fatal. 

Decision difficulties arise when goals conflict or when no good choice is available. 
For example, weather at the destination airport might be satisfactory when the plane 
takes off, but may deteriorate rapidly and be below minimums by the time the flight 
arrives. The alternat airport may have clear weather, but it may be more distant, 
straining fuel resources. All options are evaluated in terms of their level of risk, but 
sometimes no low risk option is available. Then risk must be played off against what will 
be gained in each case, factoring in the crew’s level of confidence that they can follow 
through with the choice. The crew needs to think about what might happen down the 
line. They are in a dynamic state; their equipment may be changing over time, the 
weather is changing over time and their location is changing over time. 


Faulty Risk Assessment 


Poor decisions may also arise when a flight crew is aware of conditions that require 

a decision, but underestimates the level of risk associated with the conditions, especially 
when they are changing dynamically. For example, when approaching Dallas for 
landing, the first officer of an L-1011 commented on lightning in the storm lying on 
their flight path (NTSB, 1986a). Yet, the crew flew into it and encountered wind shear. 
We know that risk is important to pilots, because potential risk was the dominant 
dimension considered by captains from several airlines when making judgments about 
flight-related decision situations (Fischer et al., 1995). Why then do crews appear to 
underestimate risk in potentially critical situations? 


162 Chapter 5 + Flight Crew Decision-Making 


One possible explanation is that crews lack the relevant experience or are unable to 
retrieve the knowledge needed to assess risk appropriately in those specific 
circumstances (cf. Klein, 1993b). Another arises from pilots’ routine experience. If 
similar risky situations have been encountered in the past and a particular course of 
action has succeeded, the crew will expect to succeed the next time with the same 
response, a phenomenon Reason (1990) called “frequency gambling.” Given the 
uncertainty of outcomes, in many cases they will be correct, but not always. Hollenbeck 
et al. (1994) found that past success influences risk-taking behavior. Baselines become 
misrepresented over time as a situation becomes familiar and the individual becomes 
more comfortable with it. Likewise, Sitkin (1992) argued that uniformly positive 
experiences provide no baseline by which to determine when the situation is becoming 
more dangerous. 


5.2.3. Plan Continuation Errors (PCE) 


Examination of decision errors in the NTSB’s (1994) analysis of 37 crew-involved 
accidents revealed an emergent theme: about 75% of the decision errors involved 
continuation of the original flight plan in the face of cues that suggested changing the 
course of action (Berman, 1995). These included taking off in snowy conditions, 
landing during an unstable approach, or continuing a VFR flight in instrument 
conditions (cf. O’Hare and Smitheram, 1995). More recent analyses confirm this 
pattern, called ‘plan continuation errors’ (Orasanu et al., 2002) or plan continuation 
biases (Dismukes et al., 2007). 

Although it is not possible to determine the cause of these patterns from post hoc 
analyses, our efforts were drawn to examining factors that might lead crews to 
demonstrate plan continuation types of decisions. In many cases it appeared that the 
crew failed to appreciate the risks inherent in the evolving conditions or those 
associated with pressing on with their original course of action. Both contextual and 
cognitive factors were hypothesized as potential contributors to these types of decision 
errors. 


5.2.4. Error Inducing Contexts 


Three types of contextual factors extracted from accident analyses may contribute to 
poor aviation decision, including PCEs: (1) poor quality information, including 
ambiguous dynamic conditions or poorly displayed information, (2) organizational 
pressures, and (3) environmental threats and stressors. 
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Information quality 


Cues that signal a problem are not always clear-cut. Poor interface design that does not 
provide adequate diagnostic information or action feedback can lead a crew astray 
(Woods & Sarter, 1998). For example, in the Kegworth crash, information about which 
engine had the problem was poorly displayed, contributing to the flight crew shutting 
down the wrong engine (AAIB, 1990). Conditions can deteriorate gradually, and the 
decision-maker’s situation awareness may not keep pace. Ambiguous cues permit 
multiple interpretations. If ambiguity is not recognized, a crew may be confident in their 
understanding of a situation, when in fact they are wrong. 

In addition to making it difficult to assess the situation, ambiguity can influence the 
decision indirectly. A crewmember may recognize that something “doesn’t seem right” 
(as stated by the first officer in the Air Florida takeoff crash in Washington, DC, during 
heavy snow with a frozen pitot tube, NTSB, 1982), but may find it difficult to justify 
a change in plan when cues are ambiguous. For decisions that have expensive 
consequences, such as rejecting a takeoff or diverting, the decision-maker may need to 
feel very confident that the change is warranted. Ambiguity thus may contribute to 
plan continuation errors. 


Organizational pressures 


An organization’s emphasis on productivity may inadvertently set up goal conflicts 
with safety. As Reason (1997) has documented, organizational decisions about 
levels of training, maintenance, fuel usage, keeping schedules, etc. may set up 
latent pathogens that undermine safety. For example, on-time arrival rates are 
reported to the public. Companies also emphasize fuel economy and getting 
passengers to their destinations rather than diverting, perhaps inadvertently 
sending mixed messages to their pilots concerning safety versus productivity. 
Mixed messages, whether explicit or implicit in the norms and organizational 
culture, create conflicting motives, which can affect pilots’ risk assessment and the 
course of action they choose. 


Environmental Threats and Stressors 


Operational factors that may affect pilots’ ability to make reasoned decisions include 
high workload, limited time, heavy traffic, poor weather, last minute runway changes, 
and schedule delays. An extensive literature documents the deleterious effects of stress 
on cognitive functioning (Hancock & Desmond, 2001; Hockey, 1979), including 
attentional focus, working memory capacity, and risk taking. These influence decision 
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process through their effects on information scanning, cue detection, hypothesis 
generation, and option evaluation. Stress also may affect crew communication, which 
can interfere with building situation models, sharing information, contingency 
planning, and error trapping. 

One of the few studies to examine stress and pilot decision making found that 
stress had little effect on decisions that drew on domain expertise and relied on 
perceptual knowledge, 1.e., recognition-primed decisions (Stokes et al., 1997). This is 
consistent with the notion that decisions are more difficult when the problem is not 
well understood and no clear response is available, i.e., in ill-defined problem 
situations. 

Certain phases of flight typically induce higher levels of stress due to heavy workload, 
traffic and little room for error recovery (Strauch, 1997), such as during takeoff and from 
top of descent to landing. Under stress, decision-makers often fall back on familiar 
responses (Hockey, 1979), but these responses may not be appropriate to the situation. 
For example, about one minute after takeoff the captain of a four-engine aircraft 
retarded power on all four engines in response to a vibration throughout the aircraft, an 
action that resulted in a crash (NTSB, 1986b). Reducing power so close to the ground 
was not appropriate because insufficient time was available for recovery. The same action 
might have been fully appropriate at a higher altitude. 

Other potentially dangerous conditions may permit more time to diagnose the 
problem and consider what to do (e.g. fuel leaks, or hydraulic, electrical or 
communication failures). However, under stress, people often behave as though they 
are under time pressure, when in fact they are not (Keinan, 1988). 


5.2.5. Cognitive Factors 


Ambiguous cues, dynamically changing risks, organizational pressures and 
environmental stressors may not in themselves be sufficient to cause poor decisions. 
However, when the decision maker’s cognitive limits are stressed, these factors may 
combine to induce errors. 


Schema-based decisions 


Consider that more than one half of the decision errors in the NTSB (1994) database 
(29 out of 51) involved omissions, or failures to do something that should have been done. 
Crews may have been captured by a familiar schema in these cases, leading them to 

do what they normally do, that is, to carry on with the usual plan, even though another 
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action was called for. Guidance by routine knowledge also supports the “cognitive 
miser” perspective: people will do as little as possible to get the job done at a satisfactory 
(though not necessarily optimal) level, Simon’s (1957) concept of “satisficing.” 


Lack of knowledge 


A cognitive economy or a schema dominance explanation, however, fails to account for 
errors of commission (NTSB, 1994). These are cases in which crews took actions that 
were out of the ordinary, such as attempting to blow snow off their aircraft using the 
engine exhaust from the aircraft ahead of them (NTSB, 1982). These cases may reflect 
“buggy” mental models or gaps in knowledge (VanLehn, 1990). Buggy models may lead 
to success in some cases, so decision-makers may have great confidence in these 
inadequate models. 

Novices are at a disadvantage in making decisions because they lack the deep and 
well-integrated knowledge of experts (Chi et al., 1988; Klein, 1998). This may be 
manifest in what appears to be inadequate situation assessment or choice of risky options. 
For example, Driskell and colleagues (1998) found that general aviation (GA) pilots only 
matched experts’ choices of options under a variety of flight conditions in terms of their 
“riskiness” 50% of the time. One reason may be their less-informed risk models. When 
Fischer et al. (2003) asked GA and commercial pilots to categorize flight scenarios on the 
basis of risk, novices focused only on the severity of the consequences, whereas experts 
included both the timeline (How long do I have to make a decision?) and controllability 
(What can I do about the situation?). 

Expert—novice differences were manifest in behaviors of more and less experienced 
pilots in several low-fidelity simulations involving deteriorating weather conditions. 
More experienced pilots made decisions earlier and traveled less into bad weather prior 
to diverting than did less experienced pilots (Wiegmann et al., 2002), suggesting 
inadequate situation awareness stemming either from lack of knowledge or different risk 
standards in the more junior pilots. 


Social factors 


Social factors may create goal conflicts that increase decision difficulty. Perceived 
expectations among pilots may encourage risky behavior or may induce one to behave as 
if one were knowledgeable, even when ignorant. For example, a runway collision in 
near zero visibility (due to fog) resulted when one aircraft stopped on an active runway 
because the crew did not realize where they were (NTSB, 1991). The captain was 
unfamiliar with the airport and was making his first unsupervised flight after a long 
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period of inactivity. The first officer boasted of his knowledge of the airport but, in fact, 
gave the captain incorrect information about taxiways. Rather than questioning where 
they were, the captain went along. 

Based on critical incident interviews with pilots flying in the extreme conditions of 
Alaska, Paletz and colleagues characterized several social phenomena that may lead crews 
into taking risks and perhaps plan continuation errors (Paletz et al., 2009). These 
include: 


m Informational social influence: accepting information obtained from another as 
evidence about reality, as in follow-the-leader behavior 


m Foot-in-the-door persuasion technique: agreement to a small request increasing 
likelihood of agreement to a large one later 


mw Normalization of deviance: an incremental acceptance of a progressively lower 
level of safety by a group of people 


m Impression management: not looking bad to themselves or to others 


m Self-consistency motives: acting in ways consistent with one’s beliefs. 


Personal Stress 


Personal stressors include concern with family matters, job security, or health issues. 
While some pilots may be able to put these matters out of mind on the flight deck, others 
may be distracted by them. These personal factors may also affect decision making by 
interfering with sleep, which can have negative effects on alertness, attentional focus, 
mood, and crew communication. I[Il-structured problems and organization-related goal 
conflicts require high levels of cognitive effort, which may be compromised in 
conjunction with other stress factors (Cannon-Bowers & Salas, 1998). Stress typically 
constrains working memory capacity (Hockey, 1979), thus limiting the decision-maker’s 
ability to entertain multiple hypotheses or to mentally simulate the consequences of 
options (Wickens et al., 1993). 


5.3. Behaviors that Characterize Effective 
Crew Decision-Making 


Behaviors associated with effective crew decision making have been identified from 
research in both high- and low-fidelity flight simulations (Orasanu, 1994), and validated 
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in actual line operations, primarily through line operational safety audits (LOSA) 
(Thomas, 2004). Both sources provide evidence of how crews manage threats, trap 
errors, and maintain positive crew climate essential to making good decisions. These 
behaviors can be broken down into taskwork skills and teamwork skills. 


5.3.1. Taskwork Skills 


Situation Awareness. Effective crews are vigilant. They monitor the environment for 
threats that may require a response, as well as monitoring progress of the flight according 
to the operative plan. They gather additional information to clarify threats. 

Build Shared Situation Models. Effective crews build shared situation models when 
threats arise. They assess and communicate the nature of the threat, the degree of risk, 
and time available. 

Update Plans. Effective crews are adaptive. They adjust to dynamically evolving 
conditions and update plans as needed (to avoid plan continuation errors). This 
includes building contingency plans to cope with uncertain situations. Threats may 
require that goals be updated to support threat management while maintaining the 
overall plan. 

Manage Tasks and Workload. Effective crews revise task priorities and reassign tasks to 
manage workload. 

Evaluate Options. Effective crews project the consequences of potential decisions to 
decide what to do. They are sensitive to competing goals and risks, such as safety, 
productivity, economic and professional consequences. 

Metacognitive Strategies. Effective crews are reflective. They check their assumptions, 
question missing information, consider what might go wrong, how likely it is and how 
serious it would be. 


5.3.2. Teamwork Skills 


Effective taskwork depends on effective teamwork. Maintaining a positive crew climate 
and trust in each other is essential for assuring that all crewmembers, especially 
junior ones, contribute to problem assessment and decision making (Salas et al., 2005). 
Trust and openness are the basis for error trapping. Accident investigations consistently 
point out the role of “monitoring-challenging” failures as links in the accident chain 
(NTSB, 1994). These failures are more frequent when the captain is the one making 
the error than when both crewmembers are responsible or when the error arises outside 
of the flight deck (Orasanu et al., 1998; Thomas, 2004). 
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Crew Climate. Positive crew climate begins with the captain’s pre-flight briefing 
(Ginnett, 1993), which establishes a climate of openness and participation. Studies show 
that effective crews are characterized by active participation of all crewmembers (Fischer 
et al., 2007; Parke et al., 2000). 

Error Trapping. When errors occur, members of effective crews are able to disrupt 
the error chain by calling out the error and correcting it (or even preventing it from 
occurring by being pro-active). Junior crewmembers are more likely to be effective in 
trapping errors made by the captain by using certain communication strategies: clearly 
describing the nature of the problem, offering a suggestion for solving it while leaving 
the decision up to the captain, and providing justification for the suggestion (why it’s 
a good idea) (Fischer et al., 2000). Challenges that are too direct or too mitigated 
(weak) are not likely to be effective, the former because they may disrupt crew climate, 
the latter because they don’t convey the seriousness of the problem. Effective 
challenges invoke a crew orientation (“we're in this together”), reflected by use of 
“we” rather than “you” or “I” in the suggestions, e.g., “We need to turn 15 degrees 
north about now.” 

Back-up Strategies. Finally, members of effective crews monitor each other for stress, 
fatigue or workload and back up each other or reassign tasks as needed. Good crews use 
compensatory strategies to manage fatigue or stress, such as double-checking 
information, status or plans (Petrilli et al., 2006). 


5.4. Can We Train Crews to Make Better 
Decisions? 


Team training approaches that focus on team process skills appear to be most effective 
in developing resilient teamwork skills essential to effective crew decision making 
(Klein et al., 2008; Salas et al., 2007). At this point, no basis exists for believing that it 
is possible to develop training to improve all-purpose decision-making skills. Decision 
strategies are learned most effectively in conjunction with domain-specific content 
(Glaser & Bassok, 1989), a reality guiding the integration of CRM skills with 
technical training for pilots under the FAA’s Advanced Qualification Program (AQP) 
(FAA, Advisory Circular #120-54A, 6/23/06). Team training approaches that focus 
on team process skills appear to be most effective in developing resilient teamwork 
essential to effective crew decision making (Klein et al., 2008; Salas et al., 2007). 
Positive evidence is accruing on the success of training in the perceptual skills and 
strategies needed for effective situation assessment (Endsley & Robertson, 2000; 
Wiggins & O’Hare, 2003). 
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5.4.1. Aviation Decision Training Models 


Several decision training models have been developed in the aviation industry, two of 
which are used by major international carriers. DODAR stands for Diagnosis, Options, 
Decide, Assign tasks, Review (Walters, 2002). FOR-DEC stands for Facts, Options, 
Risks and Benefits—Decide, Execute, Check (Hoermann, 1995). While both include 
steps of gathering information, deciding on the basis of anticipated consequences, and 
reviewing the decision, neither capitalizes on the crew’s expertise at recognizing and 
sizing up the situation, as in NDM. Both imply concurrent weighing of multiple 
options. Neither is tuned to differences in decision situations for which different 
decision strategies are appropriate (i.e. rule-based, choice and creative decisions). 
Essentially, these remain domain-independent general approaches that could be applied 
in any domain by any decision-maker. 


5.4.2. NDM-based training 


In contrast, training grounded in the NDM framework provides opportunities for 
developing rapid pattern recognition, serial consideration of options, use of mental 
simulation to evaluate options, and metacognitive skills (Cohen et al.,1996; Means et al., 
1993; Klein, 1993a). Klein (1998) has pointed out that learning to think like an expert 
involves developing deep knowledge that serves as a basis for making decisions. He 
recommends the following activities to foster this learning: 


m Engaging in deliberate practice that includes a goal and evaluation criteria 
m Building an extensive experience bank from diverse scenarios 
m Obtaining feedback that is accurate, diagnostic and timely 


m Reviewing prior experiences to derive new insights and lessons from mistakes. 


Situation assessment 


Training within an NDM framework emphasizes development of situation assessment 
skills, an element that was totally absent from traditional decision models. For aviation 
decisions, both rapid pattern recognition and diagnostic skills are needed. Recognition 
of danger cues and generation of appropriate responses to them should become 
automatic, which can only happen through repeated practice with feedback. Deliberate 
rather than recognitional situation assessment skills are required when cues are 
ambiguous, contradictory or worrisome. Risk assessment is an essential component. 
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Course of action selection 


The second major component of training within an NDM framework is evaluating 

a course of action. In most cases a workable rather than an optimal solution may be 
adequate. In the case of a rule-based or RPD decision, the option is generated upon 
recognition of the situation. Its adequacy in meeting the decision-maker’s goals is 
evaluated by using mental simulation of the likely outcome of taking the action in that 
specific context. Training needs to emphasize that evaluating one option at a time is 
appropriate under many circumstances rather than generating and assessing all possible 
options, especially under time-pressured situations. 


Metacognitive training 


Perhaps the most trainable decision-related skill complex is metacognition. Considerable 
research exists supporting the trainability of these skills across wide ranges of populations 
(Brown et al., 1986). Training strategies developed by Cohen, Freeman and Thompson 
(1998) emphasize goals, environmental conditions and actions under realistic practice 
conditions to promote accurate recognition, and repetition with feedback to facilitate 
automatic performance. It aims to sensitize trainees to domain-specific cues including 
time constraints, stakes and problem familiarity, as well as conflicts, completeness and 
reliability of information. Practice involves making metacognitive processes explicit (i.e. 
critiquing and correcting), which benefits from a team context. Devil’s advocate and 
crystal ball techniques are used to challenge assumptions and see weaknesses in situation 
assessments and plans. 

Thus, crews are prepared to cope when complications arise from multiple or 
competing goals, e.g. maintaining flight safety, saving fuel and getting the passengers to 
their destinations on time, with different options satisfying these competing goals. Crews 
learn to recognize that some level of risk always exists and that tradeoffs must be 
managed. Metacognitive assessment includes oneself: Are you fatigued or stressed? What 
are your motives in pursuing a particular option? Senders and Moray (1990) noted that 
pilots need training in “how to change one’s mind” and avoiding cognitive “lockup,” 
which may play a role in plan continuation errors. 


5.4.3. Communication training 


Build Shared Situation Models 


As unexpected dynamic conditions arise, it is essential that team members communicate to 
build a shared model of the emergent situation and how to cope with it: What is the 
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problem? What is our plan? Who does what and when? What contingencies must be 
planned for? What cues or conditions must we look out for and what will we do? Only if all 
participants have a shared model will they be able to contribute efficiently to the shared goal. 

The intent is not simply to get crews to talk more. More is not necessarily better: 
high levels of talk contribute to workload. What is desired is explicit discussion of the 
problem: its definition, plans, strategies and relevant information. Current training 
programs that are integrating CRM with technical training encourage crews to use pre- 
briefings to assure that all members know what to do in case of time-critical 
emergencies, such as how to handle aborted takeofts. 


Establish a positive crew climate through briefings 


Briefings conducted by the captain go a long way to assure that team members 
understand their role in the effort and feel comfortable offering their contributions, 
which may be critical to managing threats in challenging situations (Ginnett, 1993). 
Briefings set the tone or team climate; in both aviation and in medicine team members 
“follow the leader,” adopting the interactional style of the leader (Lingard et al., 2002). 
Salas et al. (2005) point out the importance of mutual trust in supporting mutual 
performance monitoring essential to flight safety. By establishing positive relationships, 
the leader can let the team know that she or he is not invincible and create a crew climate 
that is open and productive. 


Monitor and challenge threats and errors 


Crewmembers also must learn appropriate ways to bring problems to the attention of 
the captain (called advocacy and assertion in early CRM parlance). These include being as 
specific as conditions allow, pointing out problems, suggesting solutions and providing 
reasons for one’s concerns. Strategies identified by Fischer and Orasanu (1999, 2000) to 
be most effective in correcting errors involved crew obligation statements (such as “We 
need to deviate right about now”), preference statements (e.g. “I think it would be wise 
to turn left”) and hints (e.g. “That return at 25 miles looks mean”). In addition, requests 
that were supported by problem or goal statements (e.g. “We need to bump the airspeed 
to Vref plus 15. There’s windshear ahead.”) were rated as more effective than 
communications without supporting statements. 


5.4.4. Monitoring skills 


The above communication skills focus on how to communicate. They are predicated on 
adequate skill in monitoring threats in the environment and crew errors. Sumwalt et al. 
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(2002) recommend training for monitoring that focuses on specific areas of 
vulnerability, such as top-of-descent or points at which clearances are expected. Non- 
critical tasks should be accomplished during less critical phases. Monitoring other 
crewmembers is essential to mutual back-up behaviors. 

Effective monitoring thus depends on effective workload and task management 
strategies. Overall crew performance depends on the captain’s ability to prioritize tasks 
and allocate duties. Demands can be managed by contingency planning, but this depends 
on the captain anticipating possible problems, which in turn depends on good situation 
awareness and metacognitive skill. 


5.5. Conclusions: The Future of Aviation 
Decision-Making 


The jobs of pilots and air traffic controllers are constantly evolving. With modern 
equipment on the flight deck, pilots have more information at their disposal. Designing 
information displays to support good situation awareness and fast, accurate problem 
diagnosis is a theme of current research and development efforts. Providing information 
on risks is more problematic. New systems may be able to critique proposed plans for 
flaws—essentially, an intelligent automated metacognitive aid. Advances in technology 
will be accompanied by changes in roles and responsibilities in the not-too-distant 
future. The question will become how to prepare crews and controllers, with their deep 
knowledge and adaptability, but also with their vulnerabilities, to manage such a system. 
Mutual trust, respect, and a positive crew climate will continue to be the foundation for 
effective crew decision making in future automated systems. 
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Introduction 


Interest in Crew Resource Management (CRM) training and assessment methods has 
extended well beyond the pioneering efforts to improve aviation safety, described so 
comprehensively in the first edition of this book (Wiener et al., 1993). This chapter 
describes how a behavioral rating system (NOTECHS) designed for the assessment of 
airline pilots’ CRM skills, on a pan-European basis (Avaermaete & Kjuissen, 1998; Flin 
et al., 2003a), has been adapted for use for training and assessment of CRM skills in other 
work settings, such as acute medicine, nuclear power generation and systems 
management. 


6.1. Pilots’ Non-Technical Skills (NOTECHS) 


The term non-technical skills (NTS) is used by a range of technical professions (e.g. 
geoscientists, Heath, 2000) to describe what they sometimes refer to as “soft” skills. 
In aviation, the term was first used by the European Joint Aviation Authorities 
(JAA) to refer to CRM skills and was defined as “the cognitive and social skills of 
flight crew members in the cockpit, not directly related to aircraft control, system 
management, and standard operating procedures” (Flin et al., 2003a, p. 96). The 
CRM or non-technical skills include situation awareness, decision-making, 
leadership, teamwork, as well as being able to manage work-related stress and 
fatigue. They complement workers’ technical skills and should reduce errors, 
increase the capture of errors and help to mitigate when an operational problem 
occurs (Helmreich et al., 2003). The term non-technical skills is used in this 
chapter as a synonym for CRM skills because of its adoption by the European 
aviation authorities and subsequent application in other work domains, such as 
healthcare or business, where the term “crew” is less applicable. 

The international aviation regulators have generally mandated CRM courses (e.g. 
CAA, 2006a); consequently, the aviation industry has led the field in relation to the 
assessment of non-technical skills. The Federal Aviation Administration in the USA 
introduced the Advanced Qualification Program (AQP) in 1990s (see FAA, 2006). This 
enabled airlines to develop their own CRM training programs but they also had to 
demonstrate to the regulator that these were evaluated. In the UK, mandatory 
regulations from the Civil Aviation Authority (CAA, 2006a, 2006b) required a formal 
incorporation of non-technical (CRM) skills evaluation into all levels of training and 
checking flight crew members’ performance. This built on regulations from the JAA: 
“the flight crew must be assessed on their CRM skills in accordance with a methodology 
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acceptable to the Authority and published in the Operations Manual. The purpose of 
such an assessment is to: provide feedback to the crew collectively and individually and 
serve to identify retraining; and be used to improve the CRM training system” (JAA, 
2001, 1.965). 

This legislation resulted from a desire by the JAA to make available a generic method 
of evaluating pilots’ non-technical skills which would be applicable throughout Europe. 
The method would have to have minimal sensitivity to cultural and corporate 
differences, and be practical and effective for airline instructors and examiners. In 1996, 
the JAA Human Factors group initiated a research project, sponsored by four European 
CAAs. A research consortium consisting of pilots and psychologists from Germany, 
France, the Netherlands and the UK was established to work on the NOTECHS (Non- 
Technical Skills) study. The project team had to identify or develop a feasible, efficient 
method for assessing an individual pilot’s non-technical (CRM) skills. The system was to 
be used to assess the skills of an individual pilot, rather than a crew, and it was to be 
suitable for use across Europe, by both large and small operators, i.e. it was to be 
culturally robust. 

After reviewing behavior rating systems for pilots already being used by the larger 
European and American airlines (Flin & Martin, 2001), it appeared that none of them 
could be adopted in their original form. Nor did any single system provide a suitable 
basis for simple amendment that could be taken as an “Acceptable Means of 
Compliance” under the regulations. This was because the existing systems were either 
too complex for a pan-European basis, or too specific to a particular airline, or were 
designed to assess crews rather than individual pilots (e.g. University of Texas LLC 
(Line/Line Oriented Simulation Checklist system, Helmreich et al., 1995). Therefore, 
the project team decided that it would have to design a new taxonomy and rating 
method for assessing a pilot’s non-technical skills. 

The development method included reviews of the relevant literature and a detailed 
examination of existing behavioral marker system to assess pilot’s CRM skills. Particular 
attention was given to Helmreich et als (1995) LLC system due to its detailed and 
systematic development process. The subject matter experts who advised on the final 
design were airline captains who had considerable experience of using behavior rating 
methods. Details of the development process can be found in Flin et al. (2003a). The 
resulting NOTECHS system has four categories, each with component elements of 
behavior as shown in Figure 6.1. Examples of behavioral markers were provided to 
illustrate good and poor performance. 

Five operational principles were established with the aim of ensuring that each 
crewmember would receive as fair and as objective an assessment as possible with the 
NOTECHS system, see Box 6.1. 
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Figure 6.1 The NOTECHS system 
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A general issue about this type of skill evaluation based on observing performance is 
that judging behavior is always more subjective than judging technical facts (e.g. speed 
or flap settings). The NOTECHS rating system was designed to minimize ambiguities in 
the evaluation of non-technical skills. Several considerations were taken into account 
and made explicit in the project report (Avaermate & Kruijsen, 1998). The first related 
to the unit of observation, i.e. who is evaluated: the crew globally, the captain, or the co- 
pilot. The NOTECHS system was designed to be used to assess individual pilots. When 
an evaluation relates to individuals, a potential problem is to disentangle individual 
contributions to overall crew performance, an issue that already exists during checks 
when considering technical performance. It was argued that the NOTECHS system did 
not solve this problem in some magical fashion. Rather it was suggested that the system 
should assist the examiners to objectively point to behaviors that are related more to one 
crewmember than the other, therefore allowing them to differentiate their judgment of 
the two crewmembers. A second factor related to possible concern that raters might not 
be judging the non-technical skills on an appropriate basis. NOTECHS requires the 
instructor/examiner to justify any criticisms at a professional level, and with 
a standardized vocabulary. Furthermore, a judgment should not be based on a vague 
global impression or on an isolated behavior or action. Repetition of the behavior 
during the flight is usually required to explicitly identify the nature of the problem. 
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BOX 6.1: DESIGN PRINCIPLES FOR NOTECHS 


1. Only observable behavior is to be assessed—The evaluation must exclude 
reference to a crewmember’s personality or emotional attitude and should be 
based only on observable behavior. Behavioral markers were designed to 
support an objective judgement. 

2. Need for technical consequence—For a pilot’s non-technical skills to be 
rated as unacceptable, flight safety must be actually (or potentially) 
compromised. This requires a related objective technical consequence. 

3. Acceptable or unacceptable rating required—The JAR-OPS requires the airlines 
to indicate whether the observed non-technical skills are acceptable or 
unacceptable. 

4. Repetition required—Repetition of unacceptable behavior during the check 
must be observed to conclude that there is a significant problem. If, according 
to the JAR paragraph concerned, the nature of a technical failure allows for 
a second attempt, this should be granted, regardless of the non-technical 
rating. 

5. Explanation required—For each Category rated as unacceptable the examiner 
must: (a) Indicate the Element(s) in that Category where the unacceptable 
behavior was observed. (b) Explain where the observed NTS (potentially) 
led to safety consequences. (c) Give a free-text explanation on each of the 
Categories rated unacceptable, using standard phraseology. 


In essence, the NOTECHS method was designed to be a guiding tool to look 
beyond failure during recurrent checks or training, and to help point out possible 
underlying deficiencies in CRM competence in relation to technical failures. The 
evaluation of non-technical skills in a check using NOTECHS should not provoke 
a failed (not acceptable) rating without a related objective technical consequence, 
leading to compromised flight safety in the short or long term. In the event of 
a crewmember failing a check for any technical reason, NOTECHS can provide 
useful insights into the contributing individual human factors for the technical failure. 
Used in this way, the method can provide valuable assistance for debriefing and 
orienting tailored retraining. The prototype NOTECHS system offered a systematic 
approach for assessing pilots’ non-technical skills in simulator and in flight. Testing 
of the basic usability and psychometric properties of the NOTECHS system was 
then required, along with the influences of cultural differences. A consortium of 
European research centers and aviation companies (British Airways, Alitalia, Airbus) 
was set up in 1998 to test the NOTECHS method as the JARTEL project (Andlauer 
et al., 2001). 
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The main JARTEL study was an experimental rating task using NOTECHS based 
on eight video scenarios filmed in a Boeing 757 simulator, with airline pilots as the 
actors. The scenarios simulated realistic flight situations with predefined behaviors (from 
the NOTECHS elements) exhibited by the pilots at varying standards (“very poor” to 
“very good”). The pilots’ behaviors were rated using the NOTECHS system by 105 
instructors, recruited from 14 large and smaller airlines in 12 European countries. Each 
experimental session began with a briefing on the NOTECHS method and a practice 
session. Then, the instructors were asked to rate captains’ and first officers’ behavior in 
each of the eight cockpit scenarios using the NOTECHS score forms. The results 
indicated that 80% of the instructors were consistent in their ratings and 88% of them 
were satisfied with the consistency of the method. On average, the difference between 
a reference rating (established for benchmarking by consensus ratings in a set of trained 
expert instructors) and the instructors’ ratings was less than one point on the five-point 
scale, confirming an acceptable level of accuracy. In the evaluation questionnaire, the 
instructors were very satisfied with the NOTECHS rating system (O’Connor et al., 
2002). Cultural differences (relating to five European regions) were found to be less 
significant than other background variables, e.g. English language proficiency, 
experience with non-technical skills evaluation, and role perceptions of captain and first 
officer (H6rmann, 2001). An operational trial of NOTECHS was run with several 
airlines confirming the applicability and feasibility of the system in real check events 
(Andlauer et al., 2001). 

These first tests of the NOTECHS system showed that it was usable by airline 
instructors and appeared to have acceptable psychometric properties. It should be 
noted that these results were achieved with a minimal training period of half a day 
due to difficulties in recruiting experienced instructors to take part in the study, 
especially from the smaller companies. This level of training would be insufficient for 
using the NOTECHS system for regular training or assessment purposes. It was 
recommended that the basic training period be two full days or longer (depending on 
the raters’ previous experience evaluating pilots’ non-technical skills, see Flin et al., 
2008). 

Users of NOTECHS are expected to be certified flight instructors and authorized 
examiners, who have been trained in the application of the method for rating 
performance. NOTECHS presupposes sufficient knowledge of concepts included in the 
JAR-FCL theoretical program on human performance and limitations 
(JAR-FCL1.125/1.160/1.165—Theoretical knowledge instruction PPL/ATPL). No 
additional theoretical knowledge is required. (See CAA, 2006a and 2006b for the 
current UK position on CRM Instructors and CRM Instructor Examiners). It was 
recommended that the majority of any training should be devoted to the understanding 
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of the NOTECHS method, the specific use of the rating form, the calibration process of 
judgment and the debriefing phase. As the NOTECHS system is primarily used as a tool 
for debriefing and identification of training needs, it is important to ensure that in 
debriefing an emphasis is placed on skill components, rather than more “global” analyses 
of performance. 

In summary, NOTECHS was designed as a professional pragmatic tool for 
instructors and authorized examiners, rather than researchers (although it has been 
used for this purpose (Goeters, 2002)). It was written in common professional aviation 
language, with the primary intention of debriefing pilots and communicating clear 
advice for improvements. The preliminary evaluation of the NOTECHS system from 
the experimental and operational trials indicated that the basic psychometric properties 
were acceptable and that the method was usable and accepted by practitioners. Clearly, 
a more extensive test of the psychometric quality of NOTECHS would be desirable 
but this would require a large data set collected under standardized conditions and to 
my knowledge no project of this type has been funded. A Swiss research project used 
independent judgments from both the NOTECHS and LOSA (Helmreich et al., 
2003) systems to rate the behavior of pilots in 46 crews filmed in an Airbus 320 
simulator (Hausler et al., 2004). They concluded that “NOTECHS and LOSA give 
a very similar picture of the sample regarding the overall performance of crews that 
were rated” (Klampfer et al., 2003, p. 133). Thomas (2004) carried out an 
observational study of crews from 323 flight sectors (on Boeing 737 and Airbus 330 
aircraft) for a Southeast Asian airline. He used a set of four categories and 16 
behavioral markers adapted from the existing LOSA and NOTECHS methods to 
evaluate the crews’ non-technical performance and to compare them against errors 
and threat management. He found that across all phases of flight, crews who showed 
better decision-making skills were more likely to trap errors during the flight. In the 
pre-departure phase, an increase in crew cooperation resulted in higher levels of error 
trapping. Both situation awareness and decision-making were positively linked to 
effective threat management. In the pre-departure phase the key behaviors for threat 
management were planning and briefing. 

In response to the regulatory requirements from JAA on evaluation of CRM skills, 
many European airlines had developed their own systems by the late 1990s, such as the 
KLM SHAPE system, the Lufthansa System “Basic Competence for Optimum 
Performance” (Burger et al., 2002), and the Alitalia PENTAPERF system (Polo, 2002). 
Some of these systems made use of the basic NOTECHS framework in the design of 
their own customized systems (Hérmann et al., 2002). Other airlines initially used 
NOTECHS or their own versions of it to complement their proficiency evaluation 
methods (e.g. Finnair, Eastern Airways, Gulf Air, Iberia). The NOTECHS system offers 
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one method of ascertaining whether the CRM training provided to pilots is actually 
enhancing effectiveness of overall crew performance on the flight deck 
(Goeters, 2002). 

Formal assessment of non-technical skills using behavioral rating systems, such as 
NOTECHS, is most advanced for civil aviation pilots but more recently a number of 
other professional groups have begun to adapt a non-technical skills approach as the 
focus on competence assessment, licensing and revalidation begins to sharpen in 
healthcare and other safety-critical industries. The remainder of this chapter reviews 
some of this work and offers some general considerations for the widespread training and 
assessment of CRM skills. 


6.2. Anaesthetists’ Non-Technical Skills (ANTS) 


As concern about the rates of adverse events to patients caused by medical error grew 
(Hurwitz & Sheik, 2009; Vincent, 2006), medical professionals began to look at 
safety management techniques being used in industry. One technique that has 
attracted their interest is the training and assessment of non-technical skills (Flin & 
Mitchell, 2009). 

Some of the earliest work on CRM skills in the operating theater had been 
carried out by Helmreich, along with Swiss colleagues, who began to use 
aviation-style behavioral observations and ratings for anesthetic, surgical and nursing 
personnel. Helmreich and Schaefer (1994) observed operations using a set of nine 
categories of “specific behaviours that can be evaluated in terms of their presence or 
absence and quality...that are essential for safe and efficient function.” Their goal was 
“to develop a rating methodology that can be employed reliably by trained observers” 
(p. 242). With this instrument, they identified instances of error relating to 
inadequate teamwork, failures in preparation, briefings, communication and 
workload distribution. However, this did not provide a quantitative database to 
provide a baseline against which to measure interventions, such as training or 
organizational change. So to address this need, Helmreich et al. (1995) designed the 
Operating Room Checklist (ORCL), which consisted of a similar list of behavior 
categories with rating scales that could be used to assess the non-technical 
performance of operating teams. This was adapted from the aviation behavioral 
marker system LLC (Helmreich et al., 1995). The ORCL system was principally 
designed to measure team behaviors rather than to rate the non-technical skills of 
individual team members, such as anesthetists. 

About this time, Gaba and his colleagues were also developing CRM-type courses 
called Anesthesia Crisis Resource Management (ACRM) (Howard et al., 1992). They 
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designed one of the first behavioral rating systems for anesthetists (Gaba et al., 1998), also 
based on Helmreich et al.’s (1995) aviation system, which they used to rate anesthesia 
teams managing simulated critical events. They did not evaluate individual team 
members but the “overall non-technical performance of the primary anaesthetist” was 
rated. 

In 1999, I began to work with anesthetists Ronnie Glavin and Nikki Maran at 
the Scottish Clinical Simulation Centre and we obtained research funding to develop 
a taxonomy of non-technical skills and a method of rating them from behavioral 
observations. Like NOTECHS, this was to be based not on a team rating but on ratings 
of individual anesthetists working in a team in a simulated or a real operating theater. 
The ANTS (Anaesthetists’ Non-Technical Skills) system was developed using a similar 
design and evaluation process as we had used in the NOTECHS project. The basic set of 
non-technical skills for anesthetists was derived from the research literature, 
observations, interviews, surveys and incident analysis (Fletcher et al., 2002, 2004; Flin 
et al., 2003b). This was refined into a prototype taxonomy using panels of subject matter 
experts (consultant anesthetists). 

An evaluation of the ANTS taxonomy and behavior rating method was 
undertaken with the assistance of 50 consultant anesthetists from hospitals in 
Scotland. They were given basic training on the system and were then asked to rate 
the non-technical skills of consultant anesthetists shown in eight videotaped scenarios. 
The levels of rater accuracy were acceptable (88% to one scale point) and inter-rater 
reliability (across all 50 raters) was found to approach an acceptable level (two = 
0.56—-0.65) for a new area of evaluation and across so many raters. Given that the 
raters had no previous experience of behavior rating and minimal training (4 hours) 
in the ANTS system, it was concluded that these findings were sufficient to move to 
usability trials (Fletcher et al., 2003). The first measures of usability and acceptability 
from consultants and trainees were promising (Patey et al., 2005). The ANTS system 
has now had some preliminary trials in the UK through the Royal College of 
Anaesthetists (Glavin & Patey, 2009) and in Australia (Graham et al., 2009). It has 
been translated into German and Hebrew and has been used to evaluate simulator 
training for anesthetists in Canada (Yee et al., 2005) and in Denmark (Rosenstock 
et al., 2006). Rall and Gaba (2005, p. 3088) advised, “On the whole, the ANTS 
system appears to be a useful tool to further enhance assessment of nontechnical skills 
in anaesthesia, and its careful derivation from a current system of nontechnical 
assessment in aviation (NOTECHS) may allow for some interdomain comparisons, 
They also outline some of the general issues inherent in both technical and 
non-technical performance assessment, including criterion thresholds, how to rate 
fluctuating performance and inter-rater reliability. 
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6.3. Non-Technical Skills for Surgeons (NOTSS) 


The next professional group within medicine to become interested in non-technical 
skills was the surgeons. In one of the first research studies observing and rating the 
behavior of surgeons in the UK, Carthey et al. (2003) developed a “framework of the 
individual, team and organisational factors that underpin excellence in paediatric 
[cardiac] surgery.’ Seven non-technical skills were rated in their taxonomy at the 
individual (surgeon) level, in addition to “technical” skill. Leadership, communication 
and coordination were listed as team level behavioral markers. This tool does not appear 
to have been developed further but the authors concluded that “behavioural markers 
developed to explain aviation crew performance can be applied to cardiac surgery to 
explain differences in process excellence between surgical teams” (p. 422). 

Our research group in Scotland has now produced a taxonomy and behavioral rating 
system for individual surgeons’ non-technical skills while in the operating theater, called 
NOTSS (Yule et al., 2006b). As with ANTS for anesthetists, this was designed using 
various task analysis techniques such as reviewing the literature, observations, interviews 
and survey data, assisted by expert surgeons as subject matter experts (Yule et al., 2006a; 
Flin et al., 2006). The NOTSS system was tested in an experimental study where 
42 consultant surgeons in Scotland received basic training and then rated the behavior of 
consultant surgeons shown on videotaped scenarios as actors in a range of simulated 
cases. The scores for accuracy were acceptable and inter-rater reliability varied across 
skill categories (Yule et al., 2008b). However, it should be noted that the surgeons were 
inexperienced in rating and had not been calibrated. The NOTSS system is now being 
trialed in a number of countries including Australia, England, Japan and the USA. 
Non-technical skill sets can also be used for debriefing and reviewing participants’ 
behavior during a particular event, such as a critical incident or a “problem flight” in 
aviation (Klair, 2000). An example of using NOTSS for this purpose with a trainee 
surgeon is given in Yule et al. (2008a). 

Other teams have adapted the aviation NOTECHS system to rate surgeons’ or 
surgical teams’ behavior (e.g. Catchpole at al., 2008; Sevdalis et al., 2008). There are 
also methods being trialed for assessing operating theater teams, with observational 
rating systems such as Remote Analysis of Team Environment (RATE) (Guerlain & 
Calland, 2009) or the Observational Team Assessment for Surgery (OTAS) which 
measures five team-level skills and is based on a model of teamwork, as well as 
behaviors derived from observations and from the ANTS system (Healey et al., 2004; 
Undre et al., 2006). 

There are also some similar teamwork scales for other specialties. Thomas et al. 
(2004) devised a team rating system based on Helmreich et als (1995) behavioral 
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markers for neonatal resuscitation and compared the content of their system to several 
other medical teamwork rating scales. They used this scale to rate the team behaviors of 
132 resuscitation teams caring for infants born by cesarean section (Thomas et al., 2006). 
They found that the behaviors reflected three underlying teamwork constructs, 
communication, management and leadership, and that these were related to 
independent ratings of the teams’ quality of care (compliance with resuscitation 
guidelines). 


6.4. Scrub Practitioners’ List of Intra- 
operative Non-Technical Skills (SPLINTS) 


Surgeons while operating rely on the skills of the nurses and other practitioners who also 
“scrub” to work closely with them at the operating table (henceforth scrub nurses). As 
there was no taxonomy of non-technical skills for scrub nurses, a research project was 
established at the University of Aberdeen to identify these skills. A literature search 
identified very few studies, in fact from an initial total of 424 publications identified, 
only 13 papers had data pertaining to non-technical skills of scrub nurses (Mitchell & 
Flin, 2008). Those papers only discussed the skills relating to scrub nurses’ 
communication, teamwork and situation awareness. Semi-structured interviews with 
scrub nurses (n = 25) (mean scrub nurse experience 18 years: range 3-32 years) were 
conducted at three Scottish hospitals to extract the non-technical skills required to do 
their job effectively (Mitchell & Flin, 2009). The interview protocol consisted of general 
questions designed to elicit responses which would provide details of non-technical skills 
used in general, day-to-day working as a scrub nurse during surgery. These questions 
were designed by drawing on knowledge of the generic non-technical skill categories 
(e.g. communication, decision-making, leadership, situation awareness) which had 
emerged from previous skill taxonomy development (Flin et al., 2008). It quickly 
became apparent that the nurses were very keen to talk about their work and the 
interviews produced rich data. At the time of writing, analyses of the interview 
transcripts are ongoing. 

In order to obtain a surgical perspective on which scrub nurse behaviors assist or 
hinder the surgeon to perform his/her task, interviews were also conducted with nine 
consultant surgeons from four Scottish hospitals. The nurses’ ability to anticipate and 
hand the surgeon instrumentation in a timely fashion were skills they particularly 
appreciated: 


m “...she should watch me and be ahead of me, a step ahead...when I say knife 
she will hand me the knife and she should know what I’m going to ask next.” 
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mw “...a lot of what you need arrives in your hand without you actually having got as 
far as asking for it, it’s almost telepathy, it’s smooth, it runs.” 


m “...they [scrub nurse] don’t ask if I’m going to need a mounted suture or 
a mounted tie—it will come mounted because they know I’m working deep 
and they know TIl not be able to reach. They don’t hand me short scissors when 
I’m in the pelvis, they’re going to give me long scissors.” (Mitchell & Flin, 2009, 
p. 77). 


Surgeons do seem to prefer scrub nurses to possess a certain degree of “mindreading” 
ability although this skill appears to be a combination of knowledge of the procedure, 
familiarity with surgeons and their preferred methods and use of instrumentation. This 
knowledge, combined with the ability to listen and process sources of available 
information, e.g. conversations and monitors in the operating theater environment, 
enables them to assist the surgeon efficiently and seemingly effortlessly. These skills also 
appears to contribute to the satisfaction derived by experienced scrub nurses when 
a procedure “flows” particularly when they have planned well, have all possible 
equipment available and have anticipated the surgeon’s requirements so that he or she 
did not have to wait for anything. 

As in the development phase of previous taxonomies (NOTECHS, ANTS, 
NOTSS), the next step in the SPLINTS project is for expert panels comprising three to 
four theater nurse/practitioner team leaders to review the data segments from the 
interviews. These panels will be tasked with labeling the skill categories and also with 
providing labels for the underlying categories within those skills. It is intended that this 
project will produce a rating tool for use by scrub practitioners to reliably rate 
observations of performance in the operating theater. 

Two final examples of non-technical skills work come from the energy and banking 
industries. 


6.5. Nuclear Power Control Room Teams 


The nuclear power industry also became interested in non-technical skills and they 
developed versions of Crew Resource Management training, especially for control room 
staff who operate the reactors and other areas of the plant. Nuclear power companies 
have well-established competence assurance programs and in some countries nuclear 
regulators require control room staff to be licensed and revalidated on a regular basis. 
These assessments can include an evaluation of non-technical as well as technical skills. 
For example, British Energy has developed an assessment system which tests operators’ 
performance managing routine and emergency scenarios in the control room simulator. 
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This measures technical and non-technical skills (e.g. situation awareness, decision- 
making, cooperation) (see O’Connor et al., 2008). 


6.6. Systems Analysts: Non-Technical Skills 
for Critical Incident Management 


Global commercial organizations, such as banks, rely on large-scale, interlinked IT 
systems to support their financial and infrastructure management operations. These are 
typically built on geographically dispersed hardware platforms running shared, often- 
interdependent software systems, operated by technical support staff and engineers. 
When any kind of critical system disruption occurs requiring an immediate response 
that cannot be resolved through standard operational procedures, specialist teams of 
experts are rapidly configured to engage in a problem-solving task to provide a response 
to the situation. These experts are not co-located, so they are required to communicate 
by a teleconference facility, as well as some computer-supported networks. Depending 
on the complexity and scale of the problem, there could be more than 50 individuals on 
a teleconference (the call or bridge), phoning in from different teams and locations, 
possibly in different time zones across the globe. The person leading the call has to 
enable this very large group of specialists (interacting only by telephone) to engage in 
effective situation assessment, problem-solving and decision-making, often for complex 
technical problems. They can be working under considerable time pressure, depending 
on the system problem as well as business considerations such as proximity to financial 
markets opening or closing. 

As in many other technical professions, the training of these IT specialists is 
predominantly of a technical nature but an examination of their CIM task requirements 
revealed that these systems experts also require cognitive and social (non-technical) skills 
to work effectively on the critical incident call. While there have been a number of 
investigations into the skills required for critical incident management (Flin, 1996), there 
has been little research into the non-technical skills required for this type of critical 
incident. I was asked to work with a team of systems analysts from a major international 
bank to help them identify core non-technical skills for the task of leading these critical 
incident management (CIM) teleconferences. 

The methods used to develop their prototype CIM skill set were based on the task 
analysis techniques used in previous studies to develop taxonomies of cognitive and social 
skills for safety-critical occupations such as airline pilots, surgeons and anesthetists (see Flin 
et al., 2008). Semi-structured interviews were conducted with a number of subject matter 
experts who had experience of leading critical incident management teleconferences to 
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identify task requirements and a basic list of skills. In addition, four critical incident 
interviews (Crandall et al., 2006) were carried out with senior managers who had 
experience of CIM calls. These were recorded and then analyzed to identify effective and 
ineffective behaviors for effective incident resolution. Documentary analysis was also 
conducted using existing competence frameworks to extract a list of non-technical skills. 
Based on these sources of cognitive and social skills, I worked with a panel of systems 
experts to develop a preliminary non-technical skills framework (Coull et al., 2009) which 
has four main categories of skills, each of which is subdivided into three elements. Each 
element has examples of good and poor behaviors as shown below. 


CATEGORIES Example ELEMENT and definition 


Situation awareness Gathering information—from the application and infrastructure 
environment and people 





Decision-making Considering options—generating alternative possibilities or 
courses of action. Assessing hazards and weighing up threats 
and benefits of potential actions 





Communication and teamwork _ Establishing a shared understanding—ensuring that the team 


has necessary information to carry out the resolution, 
understand it and that an acceptable shared “big picture” of 
the case is held by team members 





Leadership Coping with pressure—retaining a calm demeanor when under 
pressure and demonstrating to the team that the situation is 
under control. Adopting a suitably forceful manner, if 
appropriate, without undermining the role of other team 
members 





The teleconference CIM skills framework is now being used as the basis of a two-day 
training course which is similar to the Crew Resource Management courses used in 
aviation. The framework has also been adapted as an individual and team debriefing tool 
for critical incidents that are managed by teleconference and the resulting data are 
analyzed for identification of strengths and limitations (e.g. possible role overload) and 
can also be used for further refinement of the taxonomy. 

For large IT teams who communicate via teleconferencing to make critical decisions 
to resolve systems problems, a key set of non-technical or CRM skills is required for 
effective task performance. This project showed that these can be identified using task 
analysis methods and the resulting skills frameworks can then be used to enhance 
training and assessment of decision-making performance in live or simulated conditions. 

To conclude this chapter, several emerging issues are considered. 
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6.7. Domain Specificity 


Non-technical skills training and assessment is likely to be most effective when it is based 
on a skills taxonomy that has been specially developed for the domain in question rather 
than simply relabeling one from another industry. At the broad category level of skills, 
e.g. leadership, decision-making, there is generally applicability across work settings but 
at the finer grained level of skills elements and actual behaviors these should be identified 
for a particular occupation. 

Moreover, ongoing domain-specific research is needed to tailor CRM programs to 
practitioners’ needs and emerging threats for their work environment. Musson (2009), 
who has an unusual combination of medical and aviation psychology qualifications, 
cautions that lists of behaviors are not enough for CRM training. “Markers represent the 
behavioural tools that good crews and good practitioners continue to use to manage and 
perform in effective teams, but behavioural markers alone do not a training programme 
make. Continually incorporated front-line data allow course designers and trainers to 
shape programmes and produce training products that are not only more effective, but 
also more likely to be perceived as effective by front-line personnel” (p. 434). 


6.8. Individuals or Teams? 


This is a perennial debate as to whether the appropriate “unit of analysis” for these 
workplace behavioral ratings is the individual or the team as a whole. The focus of the 
non-technical skills taxonomies, training and assessment tools I have been involved with 
has always been on the individual rather than the work team. In most safety-critical 
industries, work is carried out by teams of technical specialists and so the work group 
does provide the context for the individual’s behavior. However, the “unit of analysis” in 
our studies is the individual team member not the work group as a whole, because the 
individual is regarded as the basic “building block” from which teams and larger 
organizational groupings are formed. Moreover, in my experience (e.g. civil aviation, 
the energy industry, hospital medicine), people typically do not work in the same team 
every day. Due to shift and rotation patterns, on-the-job training requirements, 
organizational constraints and working time restrictions, team composition is rarely 
fixed. In the larger airlines, the same pilots rarely fly together and for that reason, the 
focus in European aviation has been on the individual pilot’s technical and non-technical 
competence, rather than on a crew and that is the approach we have adopted in 
healthcare and other industries. As shown in the early US aviation CRM courses, a key 
focus of CRM training needs to be on the pilot’s skills for forming a team on first 
acquaintance since the odds are good that the team will meet first in the cockpit. 
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6.9. Basic Teaching of Non-Technical 
Skills Concepts 


The process of introducing the ANTS system revealed some of the difficulties of 
bringing a novel type of assessment system to a profession (medicine) where there has 
been no formal assessment of competence post-qualification (Glavin & Patey, 2009). 
Not only is the notion of workplace assessment new for qualified practitioners, it 
became apparent in the early ANTS training courses for raters that the basic 
psychological language was unfamiliar to most anesthetists. For example, the term 
situation awareness was not known, although there was good conceptual understanding 
of the need to maintain attention and vigilance. This revealed a training requirement for 
basic awareness courses in non-technical skills for both ratees and raters. Pilots are taught 
and examined on the psychological and physiological factors influencing human 
performance in their initial training program (Human Performance Limitations (HPL), 
see Campbell & Bagshaw, 2002). Therefore, when pilots begin to undertake CRM 
training, they already possess the basic human factors knowledge and are familiar with 
the cognitive and social skills influencing performance. So there appears to be a need for 
ab initio CRM/HPL courses in anesthesia introducing the concept of non-technical 
skills and describing their importance for patient safety. These should focus on the non- 
technical skills required for safety in routine procedures, as well as those needed for crises 
(for which ACRM courses already exist, Howard et al., 1992). 

As with the anesthetists, it was found during the development of NOTSS that 
surgeons do not share a common vocabulary for discussing non-technical skills, nor are 
they knowledgeable about the psychological factors influencing individual and team 
performance. Consequently, CRM courses on surgeons’ non-technical skills and 
patient safety are now being delivered by the Royal College of Surgeons of Edinburgh 
(Flin et al., 2007), as well as in other countries such as England (McCulloch et al., 
2009) and the USA (Guerlain et al., 2008). As in aviation, practitioners should be 
taught the basic principles of non-technical skills and their impact on human 
performance for their own profession before they embark on CRM or other 
multidisciplinary team training. 


6.10. Not Just for Crises 


Although there are special non-technical skill sets developed for critical incident 
management, such as the Anaesthesia Crisis Resource Management courses (Howard 
et al., 1992) or the example for finance systems analysts described above, non-technical 
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skill taxonomies should be developed for the routine aspects of safety-critical jobs. 
When workers are attentive, make sound decisions, share information and cooperate 
with fellow workers, then errors and accidents are less likely to occur (see Reason, 
2008). 


6.11. Where Angels Fear to Tread 


Behavior rating systems for non-technical skills are becoming very popular, especially in 
healthcare (see Flin & Mitchell, 2009). They appear to be deceptively simple tools, 
especially when they have been carefully written in practitioner language and have rating 
systems condensed onto small booklets for use in the workplace or simulator. We have 
found it necessary to emphasize that these skill sets should be taught by practitioners who 
have appropriate training and similarly that raters need to be trained to use the related 
assessment tools. In the world of aviation, CRM trainers and examiners must be 
qualified for these tasks (CAA, 2006b). 

Finally, the experience of working on behavioral rating tools for pilots and other 
professionals’ non-technical skills has been an absorbing and at times challenging 
exercise. As an industrial psychologist, it has been a rewarding one, especially when 
practitioners become interested in the science of behavior in the workplace, appreciating 
for the very first time that psychology might indeed have some practical value. 


REFERENCES 


Andlauer, E., The JARTEL group, 2001. Joint Aviation Requirements-Translation and 
Elaboration. JARTEL Project Report to DG-TREN European Commission. Paris: 
Sofreavia. 

Avermaete, J., Kruijsen, E. (Eds.), 1998. NOTECHS. The Evaluation of Non- 
Technical Skills of Multi-Pilot Aircrew in Relation to the JAR-FCL Requirements. 
Final Report NLR-CR-98443. National Aerospace Laboratory (NLR), Amsterdam. 

Burger, K., Neb, H., Hörmann, H., 2002. Basic performance of flight crew—a 
concept of competence based markers for defining pilots’ performance profile. 
Proceedings of 25th European Aviation Psychology Conference, Warsaw. Warsaw: 
Polish Airforce. 

CAA, 2006a. Crew Resource Management (CRM) Training. Guidance for Flight 
Crew, CRM Instructors (CRMIs) and CRM _ Instructor-Examiners (CRMIEs). 
CAP 737. Version 2. Gatwick: Civil Aviation Authority. www.caa.co.uk 


197 


198 Chapter 6 - CRM (Non-Technical) Skills — Applications for and Beyond the Flight Deck 


CAA, 2006b. Guidance Notes for Accreditation Standards for CRM Instructors and 
CRM Instructor Examiners. Standards Doc. 29 Version 2. Gatwick: Civil Aviation 
Authority. 

Campbell, R., Bagshaw, M., 2002. Human Performance and Limitations in Aviation, 
third ed. Blackwell, Oxford. 

Carthey, J., de Leval, M., Wright, D., Farewell, D., Reason, J., 2003. Behavioural 
markers of surgical excellence. Safety Science 41, 409-425. 

Catchpole, K., Mishra, A., Handa, A., McCulloch, P., 2008. Teamwork and error in 
the operating room: analysis of skills and roles. Annals of Surgery 247, 699-706. 
Coull, G., Tripp, J., Davies, M., Flin, R., 2009. Critical incident management by, 
teleconference: Identifying non-technical skills. In: Wong, W., Stanton, N., (Eds.), 
NDM9. Proceedings of the 9th Bi-annual International Conference on Naturalistic, 

Making, Decision Making. London, British Computer Society. 

Crandall, B., Klein, G., Hoffman, R., 2006. Working Minds. A Practitioner’s Guide to 
Cognitive Task Analysis. Bradford, Cambridge, MA. 

FAA, 2006. Advisory Circular 120-54A. Advanced Qualification Program. Federal 
Aviation Administration: Washington. 

Fletcher, G., McGeorge, P., Flin, R., Glavin, R., Maran, N., 2002. The role of non- 
technical skills in anaesthesia: a review of current literature. British Journal of 
Anaesthesia 88, 418-429. 

Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N., Patey, R., 2004. Rating 
non-technical skills. Developing a behavioural marker system for use in anaesthesia. 
Cognition, Technology and Work 6, 165-171. 

Fletcher, G., McGeorge, P., Flin, R., Glavin, R., Maran, N., 2003. Anaesthetists’ non- 
technical skills (ANTS). Evaluation of a behavioural marker system. British Journal 
of Anaesthesia 90, 580-588. 

Flin, R., 1996. Sitting in the Hot Seat. Leaders and Teams for Critical Incident 
Management. Wiley, Chichester. 

Flin, R., Fletcher, G., McGeorge, P., Sutherland, A., Patey, R., 2003b. Anaesthetists’ 
attitudes to teamwork and safety. Anaesthesia 58, 233—242. 

Flin, R., Martin, L., 2001. Behavioural markers for Crew Resource Management: 
a review of current practice. International Journal of Aviation Psychology 11, 95—118. 

Flin, R., Martin, L., Goeters, K., Hoermann, J., Amalberti, R., Valot, C., Nijhuis, H., 
2003a. Development of the NOTECHS (Non-Technical Skills) system for assessing 
pilot? CRM skills. Human Factors and Aerospace Safety 3, 95—117. 

Flin, R., Mitchell, L., (Eds.) 2009. Safer Surgery. Analysing Behaviour in the Oper- 
ating Theatre. Ashgate, Farnham. 


CRM (Non-Technical) Skills — Applications for and Beyond the Flight Deck - Chapter 6 


Flin, R., O’Connor, P., Crichton, M., 2008. Safety at the Sharp End. A Guide to Non- 
Technical Skills. Ashgate, Aldershot. 

Flin, R., Yule, S., McKenzie, L., Paterson-Brown, S., Maran, N., 2006. Attitudes to 
teamwork and safety in the operating theatre. The Surgeon 4, 145-151. 

Flin, R., Yule, S., Paterson-Brown, S., Maran, N., Rowley, D., Youngson, G., 2007. 
Teaching surgeons about non-technical skills. The Surgeon 5, 107—110. 

Gaba, D., Howard, S., Flanagan, B., Smith, B., Fish, K., Botney, R., 1998. Assessment 
of clinical performance during simulated crises using both technical and behavioural 
ratings. Anesthesiology 89, 8—18. 

Glavin, R., Patey, R., 2009. Integrating non-technical skills into anaesthetists’ work- 
place based assessment tools. In: Flin, R., Mitchell, L. (Eds.), Safer Surgery. Ana- 
lysing Behaviour in the Operating Theatre. Ashgate, Farnham. 

Goeters, K.-M., 2002. Evaluation of the effects of CRM training by the assessment 
of non-technical skills under LOFT. Human Factors and Aerospace Safety 2, 
71-86. 

Graham, J., Giles, E., Hocking, G., 2009. Using ANTS for workplace assessment. In: 
Flin, R., Mitchell, L. (Eds.), Safer Surgery. Analysing Behaviour in the Operating 
Theatre. Ashgate, Farnham. 

Guerlain, S., Calland, F, 2009. RATE: customizable, portable hardware/software 
system for analyzing and teaching human performance in the operating room. In: 
Flin, R., Mitchell, L. (Eds.), Safer Surgery: Analysing Behaviour in the Operating 
Theatre. Ashgate, Farnham. 

Guerlain, S., Turrentine, F, Bauer, D., Calland, F, Adams, R., 2008. Crew resource 
management training for surgeons: feasibility and impact. Cognition, Technology 
and Work 10, 255-264. 

Hausler, R., Klampfer, B., Amacher, A., Naef, W., 2004. Behavioural markers in 
analysing team performance of cockpit crews. In: Dietrich, R., Childress, T. (Eds.), 
Group Interaction in High Risk Environments. Ashgate, Aldershot. 

Healey, A., Undre, S., Vincent, C., 2004. Developing observational measures of 
performance in surgical teams. Quality & Safety in Health Care 13 (Suppl. 1), 
133-140. 

Heath, C., 2000. Technical and non-technical skills needed by oil companies. Journal 
of Geoscience Education 48, 605-612. 

Helmreich, R., Butler, R., Taggart, W., Wilhelm, J., 1995. The NASA/University of 
Texas/FAA Line/LOS checklist: a behavioural marker-based checklist for CRM 
skills assessment. Version 4. Technical Paper 94-02 (Revised 12/8/95). Austin, Texas: 
University of Texas Aerospace Research Project. 


199 


200 Chapter 6 - CRM (Non-Technical) Skills — Applications for and Beyond the Flight Deck 


Helmreich, R., Klinect, J., Wilhelm, J., 2003. Managing threat and error: data from 
line operations. In: Edkins, G., Pfister, P. (Eds.), Innovation and Consolidation in 
Aviation. Ashgate, Aldershot. 

Helmreich, R., Schaeffer, H., 1994. Team performance in the operating room. In: 
Bogner, M. (Ed.), Human Error in Medicine. Lawrence Erlbaum, New Jersey. 

Helmreich, R., Schaeffer, H., Sexton, B., 1995. The Operating Room Checklist. 
Technical Report 95-10. University of Texas at Austin. 

Hormann, J., 2001. Cultural variations of perceptions of crew behaviour in multi-pilot 
aircraft. Le Travail Humain 64, 247—268. 

Howard, S., Gaba, D., Fish, K., Yang, G., Sarnquist, F, 1992. Anesthesia Crisis 
Resource Management training: teaching anesthesiologists to handle critical inci- 
dents. Aviation, Space and Environmental Medicine 63, 763-770. 

Hurwitz, B., Sheikh, A. (Eds.), 2009. Health Care Errors and Patient Safety. Wiley- 
Blackwell, London. 

Joint Aviation Authorities, 2001. JAR OPS 1.940, 1.945, 1.955, and 1.965. Hoofdorp, 
Netherlands. 

Klair, M., 2000. The mediated debrief of problem flights. In: Dismukes, K., Smith, G. 
(Eds.), Facilitation and Debriefing in Aviation Training and Operations. Ashgate, 
Aldershot. 

Klampfer, B., Haeusler, R., Naef, W., 2003. CRM behaviour and team performance 
under high workload: outline and implications of a simulator study. In: Edkins, G., 
Pfister, P. (Eds.), Innovation and Consolidation in Aviation. Ashgate, Aldershot. 

McCulloch, P., Mishra, A., Handa, A., Dale, T., Hirst, G., Catchpole, K., 2009. The 
effects of aviation-style non-technical skills training on technical performance and 
outcome in the operating theatre. Quality and Safety in Health Care 18, 109—115. 

Mitchell, L., Flin, R., 2008. Non-technical skills of the operating theatre scrub nurse: 
literature review. Journal of Advanced Nursing 63, 15—24. 

Mitchell, L., Flin, R., 2009. Theatre nurses’ non-technical skills. In: Flin, R., 
Mitchell, L. (Eds.), Safer Surgery. Analysing Behaviour in the Operating Theatre. 
Ashgate, Farnham. 

Musson, D., 2009. Putting behavioural markers to work: Developing and evaluating 
safety training in healthcare settings. In: Flin, R., Mitchell, L. (Eds.), Safer Surgery. 
Analysing Behaviour in the Operating Theatre. Ashgate, Farnham. 

O’Connor, P., Hormann, H.-J., Flin, R., Lodge, M., Goeters, K.-M., The JARTEL 
group 2002. Developing a method for evaluating crew resource management skills: 
a European perspective. International Journal of Aviation Psychology 12, 
265-288. 


CRM (Non-Technical) Skills — Applications for and Beyond the Flight Deck - Chapter 6 


O’Connor, P, O’Dea, A., Flin, R., Belton, S., 2008. Teamwork skills for nuclear 
power plant operations personnel. International Journal of Industrial Ergonomics 38, 
1028-1037. 

Patey, R., Flin, R., Fletcher, G., Maran, N., Glavin, R., 2005. Developing a taxonomy 
of anaesthetists’ non-technical skills (ANTS). In: Hendriks, K. (Ed.), Advances in 
Patient Safety: From Research to Implementation. Agency for Healthcare Research 
and Quality, Rockville, MD. 

Polo, L., 2002. Evaluation of flight crew members performance. Is evaluation 
a product or a tool? In: Truszczynski, O., (Ed.), Proceedings of the 25th European 
Aviation Psychology Conference, Warsaw. Warsaw, Polish Airforce. 

Rall, M., Gaba, D., 2005. Simulation and anaesthesia. In: Miller, R. (Ed.), Anesthesia, 
sixth ed. Churchill Livingstone, New York. 

Reason, J., 2008. The Human Contribution. Unsafe Acts, Accidents and Heroic 
Recoveries. Ashgate, Farnham. 

Rosenstock, E., Kristensen, M., Rasmussen, L., Skak, C., Ostergaard, D., 2006. 
Qualitative analysis of difficult airway management. Acta Anaesthesiology, 
Scandinavia 50, 290-297. 

Sevdalis, N., Davis, R., Koutantji, M., Undre, S., Darzi, A., Vincent, C., 2008. 
Reliability of a revised NOTECHS scale for use in surgical teams. American Journal 
of Surgery, 196, 184—190. 

Thomas, M., 2004. Predictors of threat and error management: identification of core 
nontechnical skills and implications for training systems design. International Journal 
of Aviation Psychology 14, 207-231. 

Undre, S., Sevdalis, N., Healey, A., Darzi, A., Vincent, C., 2007. Observational 
teamwork assessment for surgery (OTAS): refinement and application in urological 
surgery. World Journal of Surgery 31, 1373—1381. 

Vincent, C., 2006. Patient Safety. Churchill Livingstone, London. 

Wiener, E., Kanki, B., Helmreich, R. (Eds.), 1993. Cockpit Resource Management. 
Academic Press, San Diego. 

Yee, B., Naik, V., Joo, H., Savoldelli, G., Chung, D., Houston, P., Karatzogolou, B., 
Hamstra, S., 2005. Nontechnical skills in anaesthesia crisis management with 
repeated exposure to simulation-based education. Anesthesiology 103, 241—248. 

Yule, S., Flin, R., Maran, N., Paterson-Brown, Rowley, D., 2006a. Non-technical 
skills for surgeons in the operating room: A review of the literature. Surgery 139, 
140-149. 

Yule, S., Flin, R., Maran, N., Paterson-Brown, S., Rowley, D., 2006b. Development of a 
rating system for surgeons’ non-technical skills. Medical Education 40, 1098—1104. 


201 


202 Chapter 6 - CRM (Non-Technical) Skills — Applications for and Beyond the Flight Deck 


Yule, S., Flin, R., Maran, N., Paterson-Brown, S., Rowley, D., Youngson, G., 2008a. 
Observe one, rate one, debrief one. Using the NOTSS system to discuss 
non-technical skills with trainee surgeons. Cognition, Technology & Work 10, 
265-274. 

Yule, S., Flin, R., Maran, N., Paterson-Brown, S., Rowley, D., 2008b. Evaluating 
surgeons’ non-technical skills with NOTSS. World Journal of Surgery 32, 
548-556. 


PART 2 


CRM Training 
Applications 


Chapter 7 


The Design, Delivery 
and Evaluation of 
Crew Resource 
Management Training 


Marissa L. Shuffler, Eduardo Salas and Luiz F. Xavier 
Department of Psychology, and Institute for Simulation and Training, 
University of Central Florida 





Crew Resource Management 
Copyright © 2010, by Elsevier Inc. All rights of reproduction in any form reserved. 205 


206 Chapter 7 + The Design, Delivery and Evaluation of Crew Resource Management Training 


Introduction 


For many, Crew Resource Management (CRM) is viewed as a critical training 
component for teams and crews in high-reliability organizations. The past several 
decades have seen an explosion of CRM training programs within the aviation 
community, with major civil aviation regulators in a large number of countries now 
mandating its implementation (Merritt & Helmreich, 1996). Furthermore, the 
increasing perception of CRM as an effective tool for reducing and managing human 
error has led to its incorporation into other areas, including the oil and railroad 
industries, healthcare, and general transportation (Salas et al., 2006b). 

However, do these perceptions of CRM training effectiveness match reality? 
Although CRM has become a standard training intervention for crews and teams in 
a variety of settings, it is unclear if CRM is achieving its objectives. While CRM 
training has been designed to provide a more systematic means of ensuring crews receive 
the same type of training in team skills, a lack of a systematic approach for assessing such 
training programs has created a rather muddled picture of CRM effectiveness. Due to 
this, current CRM training evaluations vary widely in terms of what is assessed, how it is 
assessed and how this information is used to improve the delivery of training. 
Furthermore, the sheer complexity of a comprehensive CRM training evaluation can be 
daunting to many organizations. 

Therefore, the purpose of this chapter is to ease this complexity and provide a more 
uniform approach to CRM training evaluation through the presentation of a framework 
and practical guidelines that can be applied across organizations and industries. In order 
to accomplish this, we will first outline the purpose of training evaluation as it applies 
to CRM, then review current practices in CRM training evaluation, highlighting 
challenges and limitations in current evaluation techniques. Next, we will draw from the 
work of Salas and colleagues (2006a) to provide a discussion of future directions needed 
to improve CRM evaluation, built around their framework for CRM training design, 
delivery and evaluation, and incorporating a set of practical guidelines that is necessary 
for an effective CRM training evaluation. It is hoped that this approach will promote 
both advances in improving the science of training design, delivery and evaluation for 
CRM, as well as provide a foundation for practitioners to develop assessment strategies 
that are scientifically sound and functional in the real world. 


7.1. What is Training Evaluation? 


The purpose of a training evaluation is to provide a clear picture of whether or not 
a training program is in fact achieving its goals successfully. Furthermore, since CRM 
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training has been and continues to be an evolving process, evaluation accounts for these 
changes and determines if in fact such changes are effective. Assessing training is not 
a simple process. There are many components to consider in an effective evaluation, 
including the evaluation of training at individual, team and organizational levels, and the 
evaluation of both the outcomes and the program elements (Gregorich & Wilhelm, 
1993). These latter two components are perhaps the most critical, as it is important that 
an evaluation gets at not only whether learning outcomes were achieved, but also the 
elements that make a program more or less successful. 

In the training literature, these two facets are separated into training evaluation and 
training effectiveness. Although training evaluation and effectiveness are often 
considered to be one and the same, they are in fact two different concepts. According to 
Alvarez and colleagues (2004), training effectiveness is the theoretical approach utilized 
to understand the success or failure of achieving learning outcomes, whereas training 
evaluation is focused upon the methodologies designed to measure such outcomes. The 
following discussion provides a more in-depth explanation of these components and 
why both are equally valuable to evaluating CRM training. 


7.1.1. Training Evaluation 


Training evaluation can be viewed as serving three purposes: decision-making, feedback 
and marketing (Kraiger, 2002). Evaluations provide information regarding the usefulness 
and appropriateness of a program, as well as identifying the strengths and weaknesses of 
the program so that improvements can be made (Noe, 2002). Furthermore, evaluation 
results can be utilized in marketing in order to sell the program to potential trainees or 
other organizations (Kraiger, 2002). As such, training evaluation can be seen as primarily 
focused upon the learning outcomes and how their measurement can be used to benefit 
the organization, providing more of a microview for the results of training (Alvarez 

et al., 2004). 

Training evaluation is primarily conducted through the measurement of specific, 
tangible outcomes that are the desired output. Multiple models exist to define the 
best approaches to training evaluation, including the traditional model presented by 
Kirkpatrick (1976). This model is perhaps the most simplistic, as it highlights four levels 
of evaluation that should be considered by organizations conducting training 
evaluations. These levels involve reaction, learning, behavior and results outcomes, and 
will be discussed in greater detail later, as Kirkpatrick’s approach is the one most 
commonly used in CRM evaluations. 

More recent approaches have attempted to expand beyond Kirkpatrick’s original 
model in order to improve training evaluation methods. Kraiger et al. (1993) take 
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Kirkpatrick’s model a step further by detailing the outcomes (skill based, cognitive, 
affective) that must be evaluated after training. This model takes a multidimensional 
approach that is designed to provide a more comprehensive view of outcomes in order 
to best match outcomes to what is being learned. Ford et al. (in press) provide an 
additional update to this approach through their review of studies utilizing the Kraiger 
et al. (1993) model of evaluation. Ford and colleagues highlighted the need to include 
four additional evaluation methodologies that have emerged following the publication 
of the original model: mental models, metacognition, goal orientation and attitude 
strength. Such multidimensional approaches have a distinct advantage over other 
approaches to training evaluation, as they provide a more in-depth understanding of the 
specific effects of training on outcomes. 


7.1.2. Training Effectiveness 


More formally, training effectiveness can be defined as “the study of the individual, 
training, and organizational characteristics that influence the training process before, 
during, and after training” (Alvarez et al., 2004, p. 389). To determine whether training 
is or is not effective, training effectiveness takes a much broader view of assessing training 
and focuses not only on identifying whether training results in learning, but also focuses 
on identifying whether skills learned in training are actually used and transferred to the 
job. Transfer of training is a key component of training effectiveness and consists of 
determining whether skills learned in training are used on the job and maintained 
overtime (Baldwin & Ford, 1988). 

Training effectiveness is assessed not just through learning outcomes, but through 
an overall review of the training design, development and delivery process (Salas & 
Cannon-Bowers, 2001). Effective training begins with a needs analysis, conducted to 
take into account the individual differences of trainees, the organizational climate and 
the characteristics of tasks to be trained (Alvarez et al., 2004). Training will not be 
effective if it does not account for this spectrum of needs at the individual, 
organizational and task levels. In addition to needs, these levels all have unique 
characteristics that must also be accounted for when assessing training effectiveness. At 
the individual level, these characteristics involve anything the trainees bring to the 
training, including personality, motivation, attitudes, experience and expectations. 
Organizational characteristics involve those characteristics that account for the context 
of the training, including learning climate, policies, trainee selection and trainee 
notification. Finally, task or training characteristics involve the aspects of the training 
program, such as the instructional style, practice, or feedback (Salas & Cannon- 
Bowers, 2001). 
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Much of the training effectiveness literature focuses on how these factors can be 
assessed before, during and after training, as well as strategies of how to remedy factors 
that may be damaging training effectiveness, such as a lack of organizational support for 
training (Broad & Newstrom, 1992). Additionally, models of training effectiveness 
emphasize the particular characteristics that may impact learning and transfer 
performance differentially. For example, Tannenbaum and colleagues (1993) propose in 
their model that individual and training characteristics are related to cognitive learning 
and training performance, whereas organizational characteristics are more strongly 
predictive of transfer performance. These distinctions aid in understanding more 
precisely how to revise training based on the factors that impact effectiveness. 


7.1.3. Merging Training Evaluation and Training 
Effectiveness 


In summary, a successful training evaluation will involve a close look at both 
evaluation and effectiveness. While these two literatures have often been treated as 
separate entities, there has been an emerging drive to develop a comprehensive 
perspective that incorporates both areas. Alavarez and colleagues (2004) present such 
an integrated model, as can be seen in Figure 7.1. This model depicts both the 


Figure 7.1 An integrated model of training effectiveness and evaluation. 
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individual, training and organizational characteristics necessary for consideration in 
training effectiveness, as well as the multidimensional aspects of training evaluation 
that are most common across the varying models of evaluation, and those most 
consistently linked to affecting training outcomes. While this model may not 
capture every nuance of the training process, it serves as a representation of how 
effectiveness and evaluation can be integrated to successfully get at the issues critical 
to proper training evaluation. For simplicity, throughout the remainder of the 
chapter we will use the term evaluation to refer to both training effectiveness and 
evaluation. 


7.2. Why is Evaluation of CRM Training 
Necessary? 


There are several reasons why evaluation is valuable to CRM training (Salas & 
Cannon-Bowers, 2001; Goldstein, 1993). First, evaluating CRM training can aid in 
indicating if the goals of the CRM program are appropriate for achieving the 
desired outcome. This is especially important for CRM, as CRM has been used as 
an umbrella term for several training interventions. For example, early CRM 
training focused on changing attitudes and neglected training behavioral skills 
(Helmreich et al., 1999; Salas et al., 1999a). Because of this lack of attention to 
behavioral skills, these early CRM interventions were not as effective as originally 
hoped. 

More recent CRM training has focused on improving trainees’ behaviors and 
teamwork skills (Helmreich et al., 1999; Salas et al., 1999b). However, even CRM 
training evaluations that have focused on improving behavioral competencies have been 
inconsistent. For example, CRM has been used to improve communication, situation 
awareness, decision-making, leadership, preflight briefing, stress awareness, assertiveness, 
conflict management, mission analysis, among others (see Table 7.1 for more detail; Salas 
et al., 2006b). 

Second, an evaluation of CRM training can indicate if the content and methods 
used in the training result in the achievement of the overall program goals. Given the 
multitude of competencies addressed by CRM, it may be the case that while CRM 
training is effective at improving certain competencies, it is ineffective at improving 
other competencies. Thus, it is crucial for training effectiveness endeavors to clearly 
identify the specific competencies that are trained to identify what skills CRM 
training is effective at improving and what skills CRM training is ineffective at 
improving. 
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Table 7.1 Potential CRM skills to be trained 


CRM Skill 


Communication 


Definition 


Ability of two or more team members 
to clearly and accurately send and 
receive information or commands 
and to provide useful feedback. 


Alternative 
Names 


Closed-loop 
communication 


Reference 


Cannon-Bowers, 
Tannenbaum, 
Salas, & Volpc, 
1995 





Briefing 


Backup 
behavior 


Ability of team members to develop 
plans of action by organizing team 
resources, activities, and responses to 
ensure tasks are completed in an 
integrated and synchronized manner. 


Ability of team members to anticipate 
the needs of others through accurate 
knowledge about each other’s 
responsibilities including the ability 
to shift workload between members 
to create balance during periods of 
high workload or pressure. 


Mission analysis, 
Planning 


Advocacy 


Salas & Cannon- 
Bowers, 2001 


Mdntyre & Salas, 
1995; Porter, 
Hollenbeck, Ilgen, 
Ellis, West, & 
Moon, 2003 





Mutual 
performance 
monitoring 


Team 
leadership 


Decision 
making 


Task-related 
assertiveness 


Ability of team members to 
accurately monitor other team 
members performance, inducing 
giving seeking, and receiving tadk- 
clarifying feedback. 


Ability of a team leader to direct and 
coordinate the activities of team 
members, encourage team members 
to work together; asses performance; 
assign tasks; devolp team knowledge, 
skills, and abilities; motivate; plan 
and organize; and establish a positive 
team atmosphere. 


Ability of team members to gather 
and integrate information, make 
logical and sound judgements, 
identify alternatives, consider the 
consequences of each alternative, 
and select the best one. 


Willingness/readiness of team 
members to communicate their 
ideas,opinions, and observations 

in a way that is persuasive to other 
team members and to maintain 

a position until convinced by the 
facts that other options are better. 


Workload 
management 


Management 


Judgement 
problem solving 


Confidence 
Aggressiveness 
Authoritarian 





Mdntyre & Salas, 
1995; Salas & 
Cannon-Gowers, 
2001 


Cannon-Bowers 
et al., 1995; 
Zacarro Rillman, & 
Marks, 2001 


Salas & Cannon- 
Bowers, 2001 


Salas & Cannon- 
Bowers, 2001 


Continued 
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Table 7.1 Potential CRM skills to be trained—cont’d 


Alternative 
CRM Skill Definition Names Reference 


Team Ability of team members to alter a Flexibility Cannon-Bowers 

adaptability course of action or adjust strategies et al., 1995; 
when new information becomes Klein & Pierce, 
available 2001 





Shared situation Ability of team members to gather Shared mental Salas & Cannon- 
awareness and use information to devolp models, situation Bowers, 2001 

a common understanding of the assessment 

task and team environment. 





Source: (Adapted from Salas et al., 2006b) 


Not only does the content of CRM differ, but the training methods are highly 
variable as well. Different training programs have utilized lectures, discussions, 
videotapes, observations, game-playing, classroom role play, mishap analysis and high 
and low fidelity simulators (O’Connor et al., 2008; Salas et al., 1999b). Furthermore, 
some training interventions have utilized only one method such as lectures, whereas 
other training interventions have utilized a variety of methods (Salas et al., 1999a). Thus, 
an important question that needs to be answered is how much of the effectiveness of 
CRM training is attributable to the training method and how much is attributable to the 
training content. 

Finally, evaluation of CRM training programs can aid both in determining how to 
maximize the transfer of training and serve as feedback at the individual and team levels 
to provide suggestions of areas of improvement or revision. Clearly, continued 
evaluation is valuable and necessary for addressing these issues, particularly in terms of 
determining what aspects of CRM training are successful as is and which need 
improvement. 


7.3. How has CRM Training been Evaluated in 
the Past? 


In order to provide guidelines regarding future training evaluation, it is useful to identify 
what is already known about CRM evaluation. While most CRM training evaluation 
studies have been carried out in the aviation community, the popularity and apparent 
success of CRM has led other industries such as military, offshore oil production, 
nuclear power and medical to begin adopting CRM training (Salas et al., 2006a). 
Although many organizations believe CRM is effective, a key question remains 
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regarding whether CRM training actually results in learning and transfer of training. In 
the following section, we will provide a brief review of what is known regarding CRM 
training evaluation, primarily through the examination of the components of evaluation 
and effectiveness. In this discussion, we will highlight both the strengths of prior 
evaluation research, as well as some of the challenges faced. 


7.3.1. CRM Training Evaluation 


Several reviews have sought to answer the question of whether CRM is effective in 
achieving learning objectives (O’Connor et al., 2008, 2002; Salas et al., 2001, 2006a). As 
previously discussed, most of the CRM evaluation studies cited in the reviews addressed 
some aspect of Kirkpatrick’s (1976, 1987) training evaluation framework, which 
identified four levels of training evaluation. The first level, reactions, measures trainees’ 
emotional/affective responses and is mainly concerned with whether trainees liked 
training and/or found it useful. Because of the ease of collecting reaction data from 
paper and pencil surveys, reaction data is perhaps the most common form of training 
evaluation. 

The second level, learning, is concerned with whether trainees learned/absorbed the 
content, principles and facts. The third level, behaviors, is concerned with whether 
trainees can apply/use the skills taught in training on the job. Studies often conceptualize 
behaviors and transfer of training as the same thing. The last level, results, addresses 
whether training achieves organizationally relevant goals and objectives such as increased 
profit and reduced turnover. In the case of CRM, the desired training result is increased 
safety and reduced accidents. Research at each of Kirkpatrick’s four levels will be 
reviewed in the following discussion. 


Reactions 


Reaction data can be thought as the equivalent of customer satisfaction with its 
emphasis on whether trainees liked the training (O’Connor et al., 2008). The 
rationale behind collecting reaction data is the common assumption that trainees who 
disliked training are less likely to attend to training and apply trained competencies 
on the job. While those who dislike training are unlikely to apply trained skills, it 
should be noted that favorable reactions to training do not guarantee learning or 
positive transfer. 

In terms of what is known regarding the effects of CRM on trainee reactions, 
O’Connor et al. (2002) found reaction data to be the most common measure of CRM 
training effectiveness with 69% of studies in their review reporting reaction data. Salas 
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and colleagues (2001, 2006a) also found several studies reporting reaction data. In 

a review of the CRM literature between 1983 and 1999, Salas and colleagues (2001) 
located 58 studies evaluating the effectiveness of CRM training. Of the 58 studies, 27 
collected reaction data. In a follow-up to their 2001 review, Salas and colleagues (2006a) 
located 28 studies that examined the effectiveness of CRM training between 2000 and 
2006. Of the 28 studies, 13 collected reaction data. Practically all studies measuring 
trainees’ reactions reported positive results (O’Connor et al., 2002, 2008; Salas et al., 
2001, 2006a). In terms of measurement approach, almost all studies that assessed 
participants’ reactions collected reaction data via paper and pencil surveys (O’Connor 
et al., 2002), 


Learning 


While learning as conceptualized by Kirkpatrick focused primarily on the acquisition of 
factual information (i.e. cognitive learning outcomes), other researchers have proposed 
learning to be a multidimensional construct that consists of cognitive, skill and affective 
learning outcomes (Gagne, 1984; Kraiger et al., 1993). Cognitive learning outcomes are 
most closely aligned with Kirkpatrick’s conceptualization of learning outcomes and 
primarily address whether trainees acquired factual knowledge, as well as cognitive 
strategies (Kraiger et al., 1993). Skill-based learning outcomes are focused on whether 
trainees acquire necessary technical or motor skills (Kraiger et al., 1993). Affective 
learning outcomes are focused on the extent to which trainees develop the attitudes, 
motivation and goals targeted by training. 

In terms of CRM training, the research literature seems to indicate that CRM is 
effective at achieving positive learning outcomes (O’Connor et al., 2008; Salas et al., 
2001, 2006a). Studies that have evaluated the effectiveness of CRM training at 
improving cognitive learning outcomes have typically used paper and pencil measures of 
declarative knowledge. So far findings have been mixed with some studies finding that 
CRM training results in improved cognitive learning (Stout et al., 1997, Salas et al., 
1999a), while other studies show no effects on cognitive learning (Brun et al., 2000). 
Furthermore, other studies obtaining mixed findings where CRM training results in 
cognitive learning for some people and not others (Howard et al., 1992). 

Compared to cognitive and skill-based learning outcomes, affective learning 
outcomes are perhaps the most common CRM learning outcome assessed. The majority 
of studies examining attitude change find that CRM training results in positive attitude 
change (Alkov & Gaynor, 1991; Fonne & Fredriksen, 1995; Gregorich, 1993; Gregorich 
et al., 1990; Grubb et al., 1999; Irwin, 1991; Morey et al., 1997, 2002; O’Connor et al., 
2008). Most studies use the cockpit management attitudes questionnaire (CMAQ; 
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Helmreich, 1984; Gregorich et al., 1990) to measure attitude change. More specifically, 
the CMAQ measures people attitudes toward communication and coordination, 
command responsibility and stressor effects (Gregorich et al., 1990). This self-report 
measure (or a modified version) has often been used to illustrate that trainees like the 
training they received, and that they found it useful in their jobs. However, there are 
inconsistencies in terms of the linking of these self-reports to actual learning of trained 
competencies, as illustrated through the evaluation of behaviors (Salas et al., 2006a). 


Behaviors 


Behavioral data is another common method of evaluating CRM training. For CRM this 
type of data is collected through the use of the Targeted Acceptable Responses to 
Generated Events or Tasks (TARGETs) or Line/LOS checklists. TARGETs requires 
trainees to respond to multiple scripted simulated events while observers indicate 
whether trainees did or did not demonstrate targeted behaviors on a behavioral 
checklist. Studies by Salas and colleagues (Stout et al., 1997; Salas et al., 1999b) have 
utilized TARGETs to evaluate CRM training outcomes and found that individuals 
receiving CRM training demonstrated more CRM-related behaviors than individuals 
placed in a control condition. 

Unlike the relatively consistent findings for reaction and learning outcomes, 
behavioral outcomes have been more mixed. Several studies have reported positive 
results (Clothier, 1991; Connolly & Blackwell, 1987; Goeters, 2002; Grubb & Morey, 
2003; Grubb et al., 2002; Katz, 2003; Morey et al., 2002; Spiker et al., 2003). Other 
studies reported both positive and negative results (Gaba et al., 2001, 1998; Jacobsen 
et al., 2001; Roberston & Taylor, 1995; Taylor & Thomas, 2003; Taylor et al., 1993). 
Additionally, a handful of studies reported negative results (Ellis & Hughes, 1999; 
Howard et al., 1992). 

A limitation of studies collecting behavioral data is that a majority of studies collected 
behavioral data in simulators immediately after training. While this approach is useful to 
assess whether trainees can immediately apply trained behaviors, it fails to provide data 
regarding whether trainees actually use training on the job. Different approaches to 
address this limitation will be addressed later in the chapter. 


Results 


The results level of CRM training evaluation has been the least studied of Kirkpatrick’s 
four levels. Both the complexity of obtaining such data as well as the extensive resources 
required (as compared to other levels of evaluation) are primarily responsible for this lack 
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of more complete evaluations. This low number is reflected in recent meta-analyses 
conducted on CRM training evaluation, with Salas and colleagues (2006b) reporting that 
only five out of 28 studies collected data at the results level of analysis. Similarly, Salas et al. 
(2001) reported only six out of 58 studies collected data at the results level of analysis. 
Based on this limited sample size, it appears that CRM training produces positive 
organizational results such as reduced accidents and injuries (Diehl, 1991; Kayten, 1993; 
Grubb & Morey, 2003; Taylor et al., 1993). While these findings are promising, one 
should be cautious about generalizing these findings because they are based on only 
a handful of studies. Furthermore, the challenges of evaluating training at this level 
(e.g. lack of control over extraneous variables, difficulty identifying criterion measures) 
provide additional hesitation in concluding that CRM training is in fact directly linked 
to such outcomes. 


7.3.2. CRM Training Effectiveness 


As illustrated through the lack of evidence for the results stage of Kirkpatrick’s (1976) 
training evaluation model, most CRM studies have focused on training evaluation and 
have neglected training effectiveness. Training transfer is a key component of training 
effectiveness. Unfortunately, few studies have examined whether CRM skills are 
successfully transferred from training onto the job (Salas et al., 2001, 2006a; O’Connor 
et al., 2008). 

As previously discussed, training effectiveness is also concerned about the individual, 
training and organizational characteristics that influence learning and training transfer. 
Individual characteristics that can affect CRM training effectiveness include intelligence, 
motivation, self-efficacy and organizational commitment (Gregorich & Wilhelm, 1993). 
However, the specifics of how these characteristics impact CRM training effectiveness, 
including what point in the training cycle they are most influential, is currently 
unknown and in need of future research. 

Training characteristics include CRM training design elements that can impact 
training effectiveness such as information delivery (i.e. lecture, self-paced readings), 
practice elements (i.e. role plays and simulations) and training materials (i.e. outlines, 
handouts; Beard et al., 1995). Although there are several federal regulations regarding 
the requirements of CRM training, none describe the appropriate design elements such 
as training methods or strategies. 

Therefore, a range of these design elements have been incorporated into CRM 
training, with some being more successful than others. In particular, evaluations show 
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that low fidelity simulations can be used to effectively train CRM-related skills (Bowers 
et al., 1992; Baker et al., 1993; Jentsch & Bowers, 1998). Additionally, training 
evaluations have illustrated the importance of scenario design (Prince et al., 1993; Prince 
& Salas, 1999) and scenario feedback (Salas et al., 2000; Prince et al., 1997). However, 
while there is evidence suggesting how these training characteristics may impact CRM 
effectiveness, much is still to be learned, as there are many other training design aspects 
of CRM that have not been effectively evaluated. 

The final aspect of training effectiveness involves organizational characteristics. 
Organizational characteristics are concerned with system-wide factors that can 
affect training effectiveness such as supervisor support, rewards for utilizing trained 
skills and organizational climate that values using CRM. Unfortunately, much like 
assessing training transfer, few studies have gathered and/or reported data regarding 
individual, training and organizational characteristics that affect training 
effectiveness. 

A primary issue that has been noted, however, is the effect of organizational culture 
on CRM training. Evaluations of the first several iterations of CRM training found that 
those designed for specific organizations did not transfer well to other organizations, 
emphasizing the need to develop CRM that meets organizational needs (Helmreich 
et al., 1999). Future research is needed to further explore additional organizational 
characteristics such as these, as they may have a significant impact on the effectiveness of 
CRM training. 


7.3.3. Summary 


A summary of this prior research on CRM training evaluation and effectiveness can 
be found in Table 7.2. After almost three decades of research, much remains 
unknown regarding the evaluation of CRM training. What is known, however, is 
that from prior evaluations, CRM seems to produce positive reactions, learning 
outcomes (both knowledge and attitude change) and behavior change. However, it 
remains unknown if CRM achieves organizational level results such as improved 
safety. Furthermore, less is known regarding the effectiveness of CRM training, as few 
studies have been conducted to specifically examine the individual, training and 
organizational characteristics that impact CRM training. Since the primary goal of 
CRM is to improve safety and reduce error, it therefore remains imperative for 
evaluation of such programs to continue, especially to link CRM more clearly to such 
outcomes and address potential characteristics that may detract from training 
effectiveness. 


218 Chapter 7 - The Design, Delivery and Evaluation of Crew Resource Management Training 


Table 7.2 Summary of CRM training evaluation 


Evaluation 
criteria 


Reactions 


Learning 


Behaviors 


Key findings 


Reaction data are the most common measure of 
CRM training evaluation 

Majority of studies assess reactions using paper and 
pencil surveys 

Practically all studies measuring trainee reactions to 
training reported positive results 


Cognitive, skill and affective learning outcomes are 
all evaluated in CRM training, with affective being 
the most common 

Majority of studies involve assessing learning 
through declarative knowledge tests for 

cognitive and skill 

learning outcomes, CMAQ for affective learning 
outcomes 

Generally, CRM training results in positive attitude 
change 

Findings for cognitive and skill learning are more 
mixed, as some studies found that CRM training 
improved cognitive learning, while others found 
no effects 


Behavioral data are collected through the use of 
TARGETS or Line/LOS checklists 

Behavioral outcomes are mixed, with studies 
reporting positive results, a mix of negative 

and positive results, or just negative results 

A limitation of behavioral data is that they are 
often collected in simulators immediately after 
training, which fails to capture long-term 
impacts on transfer 

of training 


Example sources 


Brun et al., 2000; 
Taylor, 1998; 
O’Connor et al., 
2002, 2008; Salas 
et al., 2001, 2006a 


Alkov & Gaynor, 1991; 
Fonne & Fredriksen, 
1995; Gregorich, 
1993; Gregorich et al., 
1990; Grubb et al., 
1999; Irwin, 1991; 
Morey et al., 1997, 
2002; O'Connor et al., 
2008 


Clothier, 1991; 
Connolly & Blackwell, 
1987; Goeters, 2002; 
Grubb & Morey, 
2003; Grubb et al., 
2002; Katz, 2003; 
Morey et al. 2002; 
Spiker et al., 2003; 
Jacobsen et al., 2001; 
Roberston & Taylor, 
1995; Taylor & 
Thomas, 2003; Taylor 
et al., 1993; Ellis & 
Hughes, 1999; 
Howard et al., 1992 





Results 


Results data are the least studied of the four 

levels of evaluation 

CRM training has been shown to reduced accidents 
and injuries, but the limited sample size provides 
caution in interpreting these results 

Challenges of collecting data at this level 

include the lack of control over-extraneous 
variables and difficulty identifying criterion 
measures 


Diehl, 1991; Kayten, 
1993; Grubb & Morey, 
2003; Taylor et al., 
1993; Salas et al., 
2006a 
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Table 7.2. Summary of CRM training evaluation—cont’d 


Evaluation 
criteria 


Training 
transfer 


Key findings 


Few studies have focused on the transfer of CRM 
training onto the job 

Transfer of CRM training has primarily been assessed 
through simulations and videotaped observations 

of participants engaged in scenarios 

Mixed results have been found when assessing training 
transfer 


Example sources 


O'Connor et al., 2008; 
Salas et al., 2001, 
2006a; Spiker et al., 
1999 





Individual 
characteristics 


Training 
characteristics 


Individual characteristics that affect CRM training 
effectiveness include motivation, intelligence, 
self-efficacy, organizational commitment and personality 
However, the specifics as to how these 

characteristics impact CRM training have not been studied 


Training characteristics that impact CRM training 
include information delivery, practice elements 

and training materials 

Low fidelity simulations have been found to 
effectively train CRM skills 

Scenario design and feedback are also an important 
consideration for CRM training effectiveness 


Gregorich & Wilhelm, 
1993; Gregorich et al., 
1990; Helmriech 

et al., 1989; Cannon- 
Bowers et al., 1989 


Beard et al., 1995; 
Bowers et al., 1992; 
Baker et al., 1993; 
Jentsch & Bowers, 
1998, Prince et al., 
1993; Prince & Salas, 
1999 





Organizational 
characteristics 


The impact of organizational characteristics on CRM 
training effectiveness is rarely assessed 
Organizational and national culture has been shown 
to impact CRM training, with organization-specific 
training not transferring well to other organizations 


Helmreich et al., 
1999; Gregorich & 
Wilhelm, 1993; 
Helmreich & 
Merritt, 1998 





7.4. How Should CRM Training be Assessed 
in the Future? 


As the previous section highlights, there are multiple challenges to the effective 
evaluation of CRM training. However, this is not to say that it cannot be done; instead, 
it is important that those assessing CRM training ensure that the proper procedures are 
followed in order to develop the most effective evaluation possible. In the following 
section, we will build upon the past challenges of CRM evaluation as well as draw from 
current knowledge of the science of training in order to provide a clear discussion of the 
considerations that must be made when developing a CRM evaluation. To do so, we 
draw from the previous work of Salas and colleagues (2006a), highlighting their model 
of CRM training design, implementation, evaluation and transfer (see Figure 7.2) to 
identify areas of particular need of attention in future evaluation. Furthermore, we 
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Figure 7.2 The progression of CRM training design, development and evaluation. 


CRM Training Needs Analysis CRM Training Design CRM Training Development 


Uncover the necessary CRM Rely on the science of learning and Specify specific CRM learning 
competencies. training. opportunities. 
Develop CRM training Goals. Develop CRM training objectives. Develop full-scale prototype of 
Ensure the organization is ready Determine what to measure and training. 
for CRM training. how to measure it. Validate and modify CRM training. 


CRM Training CRM Training CRM Training CRM Training 
Implementation Evaluation Transfer Outcomes 


Prepare trainees and the Determine training Establish a climate for Safety 
environment. effectiveness. transfer. Improved productivity 
Set a climate for learning Evaluate CRM training at Reinforce CRM Fewer errors 
(e.g., practice and feedback). multiple levels. behaviors on the job. Improved teamwork 
Implement the CRM training Revise the CRM training Provide recurrent CRM Better decisions 
program. program to improve training. 
effectiveness. 


Source: (Adapted from Salas et al., 2006a) 





intersperse throughout this section several guidelines that should be considered for 
effective CRM evaluation. It is hoped that this information will serve to better inform 
both the science and practice of CRM training and evaluation. 


7.4.1. A Framework for CRM Training Design, 
Development and Evaluation 


As previously discussed, there are several areas in need of attention regarding 
CRM evaluation. First and foremost, however, is the need for a systematic tool to 
assess CRM training. While previous models have been developed to address this 
issue (e.g. Gregorich & Wilhelm, 1993), recent advances in the science of training 
have led to the need for a revised approach. Of particular interest is the 
progression of CRM training and its related checklist as provided by Salas and 
colleagues (2006b). 

Recognizing the overall lack of consistency in CRM trainings, Salas and colleagues 
(2006b) identified both a scientifically based progression of CRM training (see 
Figure 7.2) and an in-depth checklist (see Table 7.3 for an abbreviated version) that can 
be used to guide CRM training developers throughout the entire process, from design to 
evaluation. As a detailed explanation of the entire progression and checklist is beyond 
the scope of this chapter, we highlight the components of each that are relevant to the 
future of CRM evaluation in the following discussion, paired with recommended 
guidelines for each component. 
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Table 7.3 CRM training assessment checklist 


Step 


Considerations 


V. CRM Training Evaluation 





Evaluate CRM 
training program. 














Revise CRM 
training program. 











Have both utility and affective reaction data (i.e., 
attitudes) been collected? 

Has learning been assessed at multiple levels? 

Has behavior been assessed in a transfer situation? 
Has the impact of training on the organization been 
evaluated at multiple time intervals 

(e.g., immediately, 3 moths after, 6 moths after)? 
Have the data been analyzed to determine 
instructional effectiveness? 


Are any revisions needed based on the 
empirical data? 

How will the revisions be implemented? 
What impact will the revisions have? 

Are the revisions cost-effective? 

How long will the revisions take? 

How will it affect upcoming training sessions? 


Outcome 


Data on CRM 
training’s 
effectiveness are 
collected at four 
levels. Data on job 
performance 

are collected. 


CRM training is 
revised on the 
basis of empirical 
data. 





VI. CRM Training Transfer 








Establish the 
climate for transfer. 











Is there supervisor support? 

Are the resources available to support the 
transfer of knowledge and skills? 

What rewards system is in place? 

Are trainees encouraged to learn form mistakes? 


Supervisors support 
CRM competencies 
on the job. 
Organization 
supports CRM 
competencies on the 
job. Continuous 
learning climate is 
established. 








Reinforce RM 
behaviors. 














Provide recurrent 
CRM training 











Are trainees being rewarded to encourage the 
transfer of the trained CRM competencies? 
Are behaviors that contradict what was taught 
in CRM training discouraged? 


How often does training need to be offered? 





Source: (Adapted from Salas et al., 2006b) 


Trainees are 
rewarded. Behaviors 
that contradict CRM 
are discouraged. 


CRM competencies 
remain stable over 
time. 


7.4.2. Guidelines for CRM Training Evaluation 


In terms of the progression of CRM training as recommended by Salas and 
colleagues (2006a), it is important to note that CRM training evaluation begins not 
just with the training evaluation, but with the initial training needs analysis. As 
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previously discussed, to thoroughly assess the effectiveness of an organization, it is 
important that the individual, training and organizational needs are identified and 
addressed throughout the training design and development process. A needs 
assessment consists of collecting information regarding where training is needed, 
what needs to be trained and who needs to be trained (Goldstein & Ford, 2002). The 
three essential components of a needs analysis are task, person and organizational 
analysis. Task analysis provides information regarding task duties and difficulty. 
Organizational analysis provides information regarding organizational-level factors 
that can affect training effectiveness such as organizational culture, social support and 
strategic objectives. Person analysis gathers information regarding the personality 
characteristics, adaptability, tolerance for ambiguity, and strengths and weaknesses of 
individual employees. An effectively designed needs analysis will enable trainers to 
identify what skills need to be trained, which employees will benefit most from 
training and what organizational barriers may hamper training effectiveness. The 
needs analysis serves as the foundation upon which a training evaluation can be built, 
and is therefore critical to the evaluation process. 


Guideline 1: A needs analysis is the foundation of CRM 


training evaluation 


Moving from the needs analysis phase of training, we next discuss the implications of 
training design on CRM assessment. As mentioned previously, CRM has been used as 
an umbrella term to cover a range of competencies and training methods. Several 
studies fail to mention the competencies targeted by their CRM intervention. 
Additionally, several studies fail to mention the specific methods used to train CRM 
competencies. In order to properly assess training effectiveness, both the training 
content and methods must be clearly operationalized. By doing so, evaluation criteria 
can be derived from and built upon the specific training objectives, content and 
methods, ensuring that what is evaluated is actually relevant to the objectives and goals 
of the training program. Furthermore, this will allow other trainers and researchers to 
compare the effectiveness of CRM training programs based on their target 
competencies and training methods. 


Guideline 2: Build evaluation criteria using CRM-based 
learning outcomes 


The next phase of Salas and colleagues’ (2006a) progression of CRM training that is 
relevant to assessment involves training evaluation. Several training researchers have 
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recognized the limitations of Kirkpatrick’s four levels of training evaluation (Alliger & 
Janak, 1989; Alliger et al., 1997; Alvarez et al., 2004; Kraiger et al., 1993; Holton, 1996). 
Despite frequent calls for trainers to move away from Kirkpatrick’s four levels, this 
model of training evaluation continues to be the most frequently used approach. We too 
echo the call for training practitioners and researchers to move away from using 
Kirkpatrick’s four levels. At the very least, trainers should be aware of the limitations 
associated with Kirkpatrick’s four levels. More specific issues associated with 
Kirkpatrick’s model and recommendations to overcome those issues will be discussed 
in guidelines 4-9. 


Guideline 3: Move beyond Kirkpatrick’s levels 
of evaluation in CRM assessment 


The over-reliance on reaction data being the primary form of CRM training 
effectiveness is problematic because reaction data have been found to be unrelated to 
learning, behaviors and results (Alliger et al., 1997). If CRM training is evaluated solely 
at the reaction level, training practitioners and researchers cannot be confident that the 
training program is effective at meeting organizational goals. We are not arguing for the 
complete abandonment of collecting reaction data. If trainees feel that a training 
program is useless, it is beneficial to know their feelings in order to modify training to be 
viewed more favorably. However, it is important for evaluators to recognize the 
limitations of reaction data. Additionally, if reaction data are collected, collecting utility 
reactions (i.e. how useful was training?) is more useful than collecting affective reactions 
(i.e. did you enjoy training?) because utility reactions have been found to be related to 
learning and transfer (Alliger et al., 1997). 


Guideline 4: Utility reactions should be used to supplement 
training evaluations, not serve as a cornerstone for basing all 
assessments 


Salas and colleagues (Alvarez et al., 2004; Kraiger et al., 1993) argue for 

a multidimensional view of learning that consists of affective/attitudinal, behavioral/ 
skill-based and cognitive learning. However, Kirkpatrick’s model is largely focused on 
cognitive learning. Training evaluators must be aware of the importance of collecting 
measures of different learning outcomes because each aspect of learning may be related 
to the successful enactment of CRM behaviors. To date, CRM training assessment 
studies have done a good job of measuring attitude change through self-report responses 
to the CMAQ and skill-based learning through trainees’ performance in simulator 
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exercises. However, more studies need to include measures of cognitive learning. This 
can easily be accomplished by including a paper and pencil measure of declarative 
knowledge. 


Guideline 5: When evaluating learning, measure more than 
just declarative knowledge 


The third level of Kirkpatrick behaviors (i.e. transfer of training) can be viewed as one of 
the primary goals of training. Few CRM training assessment studies have evaluated 
training transfer. The Line Operations Safety Audit (LOSA) is an assessment technique 
that may be promising to assess training transfer. The LOSA technique involves having 
trained observers ride in cockpits and observe pilots in action. Additionally, observers 
can interview pilots during and after the flight to gather additional information. 
Alternatively, trainees can be asked to complete simulator exercises, knowledge tests, or 
report on how often they utilized trained skills on the job after the passage of sufficient 
time (i.e. 6-12 months) to see if training is retained and used by trainees. Unfortunately, 
transfer of training is often left out of evaluation, as its collection involves more extensive, 
longitudinal assessment. However, it is critical to truly understanding the effectiveness of 
a CRM training program, and therefore must be addressed in assessment as such. 


Guideline 6: Don’t forget to measure training transfer 


In Kirkpatrick’s model, behaviors are primarily determined by learning. However, 
extensive research on training transfer has revealed that transfer is influenced by 

a wide range of variables besides learning such as organizational/environmental 
characteristics, individual characteristics and training design characteristics (Baldwin & 
Ford, 1988). In order to predict training transfer, training evaluators must collect 
organizational/environmental, individual and training design data. This is typically 
accomplished by carrying out a well-designed needs analysis (Alvarez et al., 2004). 
By collecting organizational/environmental, individual and training design, 

training evaluators can better identify factors that may impede the effectiveness of 
training. 


Guideline 7: Be sure to measure individual, training and 
organizational characteristics 


Despite its limitations, one key takeaway from Kirkpatrick’s model is the recognition of 
the importance of collecting multiple measures of training effectiveness at multiple 
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levels. Additionally, several calls have been made for the use of multiple measures of 
training effectiveness when evaluating CRM training (Salas et al., 1999, 2006a). Due to 
the relative ease of collecting reaction and learning data, evaluators should include them 
when assessing training effectives. To be most useful, reaction data should ask trainees if 
they find training useful. Multiple assessments of learning should be included. While the 
CMAQ has been useful as a measure of attitudinal learning, a major limitation of the 
CMAQ is that it only measures interpersonal attitudes. The CMAQ fails to address 
attitudes toward cognitive aspects of CRM such as decision-making and situation 
awareness (O’Connor et al., 2008). Efforts should be made to measure both 
interpersonal and cognitive attitudes. Additionally, evaluators must make more attempts 
to measure cognitive learning to insure that trainees actually learned key training 
concepts. Measures of cognitive learning can be easy to construct such as paper and 
pencil measures of declarative knowledge. Simulators can be used as both a measure of 
skill-based learning and training transfer. One well-tested simulation methodology that 
has been used to evaluate the effectiveness of CRM training is the Targeted Acceptable 
Responses to Generated Events or Tasks (TARGETs) method. 


Guideline 8: Collect multiple measures of training 
effectiveness at multiple levels 


Moving on to the training outcomes phase of the Salas et al. (2006b) model, we 
recognize that an important facet of outcome data is assessing the effects of training on 
outcomes over time. Several calls have been made for evaluators to collect longitudinal 
data (Brannick et al., 2005; Salas et al., 2006a). Collecting longitudinal data is important 
because training skills have been found to decay with the passage of time. With the 
majority of studies measuring behaviors in simulations, more field observations are 
needed in order to determine if CRM skills are actually being used by pilots during 
actual flights. At the very least, pilots should participate in simulators several months after 
training to determine if trained skills are maintained over time. Assessing knowledge and 
attitudes over time is comparatively easier. Former trainees can be given a short 
knowledge test, as well as the CMAQ at a specified interval of time after training. 


Guideline 9: Collect longitudinal data at the individual and 
organizational levels to assess effects of training over time 
As a final consideration, organizations must be prepared to devote adequate resources to 


the entire training design, delivery and assessment process. In order to properly assess the 
effectiveness of CRM training, organizations must acknowledge the importance of 
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gathering training effectiveness data. Part of recognizing the importance of CRM 
training involves providing trainers with the necessary resources (i.e. funding and time) 
to carry out a well-design training assessment (Salas et al., 2006b). Resources are 
especially needed if trainers attempt to evaluate training at multiple levels and/or 
longitudinally. In addition to providing trainers the necessary resources, organizations 
must champion the importance of evaluating training. A supportive organizational 
climate that recognizes the benefits of both training and its assessment is critical. In order 
to encourage employees, managers and supervisors must publicly support training 
evaluation efforts. If supervisors and managers do not support assessing training 
effectiveness, trainers may have considerable difficulty convincing people to participate. 


Guideline 10: Devote adequate resources and support 
to training assessment 


7.4.3. Summary 


While these guidelines are not necessarily the only set of practices necessary for 
successful CRM assessment, they do highlight critical areas in need of attention for an 
assessment to be effective. Additionally, if CRM trainers follow these guidelines 
consistently, comparisons across training programs will be more effective as well, as the 
same types of data should be available across assessments (e.g. multiple source, multiple 
level, longitudinal). Finally, it is hoped that providing this guidance to organizations 
implementing CRM training programs will aid in improving the process so that the 
critical error management goals can be reached and sustained over time. 


7.5. Concluding Remarks 


CRM training is a continually evolving process. As such, a systematic approach to the 
assessment process is necessary to ensure that these evolutions and changes continue to 
produce effective training programs. While prior research has illustrated the value of 
assessment, less has been provided regarding how to approach assessment in a way that is 
both scientifically driven and practically based. It is hoped that the framework and 
guidelines presented within the current chapter will serve this purpose, providing 
practitioners with general practices that can be implemented across a range of 
organizations and industries. CRM training is only as good as its results, and thus 

a systematic approach such as this can aid in ensuring that the appropriate steps are taken 
to obtain the best results possible. 
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Introduction 


The arrival of jet transport aircraft in the 1960s heralded a rapid growth in air travel, 
increase in the public’s risk exposure, and eventually a revolution in the way airlines are 
operated and crews are trained. Rudimentary ground-based artificial trainers were used 
extensively in training World War II aviators. The use of synthetic trainers in civilian 
aviation increased as the fidelity of simulation improved with technological advances and 
the cost of training in aircraft exploded with the growing sophistication of the airplanes. 
Early simulation was almost exclusively directed toward development and testing of 
individual pilots’ technical skills, the “right stuff’? Acceptable air safety was only attained 
when training and checking processes specifically incorporated “human factors” and 
crew (“team”) skills as primary considerations along with technical proficiency. 

The importance of effective teamwork in aviation cannot be overemphasized. The 
failure of the flight crew to perform as an effective team may lead to a loss of life. 
Hackman (1993) notes that “it is the team, not the aircraft or the individual pilot, that is 
at the root of most accidents and incidents.” 

Traditionally, pilot training has concentrated mainly on the development of the skills 
of the individual pilot, on individual performance. Indeed, both researchers and prac- 
titioners suggest that more emphasis should be placed on the performance of the crew as 
a team and on factors that affect crew coordination and teamwork (Hackman, 1993; 
Johnston, 1993). Diehl (1991) further points out that ideally team skills and the prin- 
ciples of Crew Resource Management (CRM) need to be introduced earlier, contin- 
uously reinforced and reviewed during flight training. Similarly, Johnston (1993) stresses 
that “if we want pilots to perform as a crew—as team members—we should train them 
as a crew throughout.” 

The data are compelling. The most common cause of an airline accident is 
Controlled Flight Into Terrain (CFIT). The inability of flight crews to communicate, 
manage workload and to maintain situational awareness is the root cause of 70% of 
accidents (Figure 8.1). 

However, human factors and Crew Resource Management (CRM) team training 
was slow to take off in the aviation community. The main reason was a cultural paradox, 
which is comprised of clusters of subparadoxes. Most airlines required their personnel to 
be exposed to human factors and team training, thus tacitly acknowledging that these 
issues are important. Yet most organizations operate in a profit-motive environment, 
which often leads to decisions being made that are antithetical to the principles of 
human factors and team training. For example, it is well established that human factors 
and team dynamics are fundamental approaches to life and work, yet most organizations 
provide one short exposure to these issues and expect real change to occur. That is, there 
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Figure 8.1 The advanced qualification Program Philosophy was originally devel- 
oped for team CRM training and assesment 
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is an expectation that life-long habits can be changed by an intervention lasting a day 
or two. 

To address this organizational challenge, fundamental human factor instructional 
requirements must be combined with interdisciplinary team training, and integrated 
throughout the education of airline personnel. This training is developed by curriculum 
designed from a task analysis of the team training requirements. There should be two 
distinct components for this training. The initial and continuing education of airline 
professionals is essential to reinforce and maintain these critical skills. This training 
becomes an integral component of the assessment criteria, and includes interdisciplinary 
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simulations that incorporate both technical and interpersonal skills designed into 
dynamic scenario presentations. 

The second paradox regarding how we train teams is the challenge introduced when 
several different disciplines must come together on a functional team. 

In aviation we have learned the need to train across disciplines. On March 9, 
1989, an Air Ontario Fokker F-27 was getting ready for takeoff from a small airport 
in Northern Ontario. There were two pilots, two flight attendants and 64 passengers 
on board including two commercial pilots traveling with their families. Takeoff was 
delayed as the tower waited for a small private aircraft to land. It was lost in the spring 
snowstorm. While the jet waited for takeoff clearance, several passengers took note of 
the accumulation of snow on the wings. One of them brought it to the attention of 
the in-charge flight attendant who assured him that there was nothing to worry 
about. The flight attendants thought it appropriate not to say anything to the 
operating pilots. 

The aircraft took off and crashed in a wooded area just beyond the runway because of 
the snow on the wings. There were 24 fatalities including the two operating pilots and 
one flight attendant. 

When asked about this during the course of the investigation and subsequent public 
inquiry, the one surviving crewmember, a flight attendant, stated that she did not feel it 
was her job to inform the pilots of a potential problem. She had never been trained to 
question an area that in her mind was clearly a pilot responsibility. 

A British Midlands 737 crashed near Kegworth in Leicestershire. To quote from the 
AIB’s Aircraft Accident Report (1989) “Although the cabin crew immediately became 
aware of heavy vibration at the onset of the emergency and three aft cabin crew saw 
flames emanating from the No 1 engine, this information was not communicated to the 
pilots.” 

Many more incident/accidents can be attributed to the fundamental breakdown in 
teams across disciplines in aviation. These comments are not intended to apportion 
blame to any of the crewmembers involved in these accidents. What was identified was 
that their training and procedures did not give them the tools to operate any other way. 
A study conducted at the NASA Ames Research Center in California identified the 
following five factors as influencing the differences between the two disciplines which 
contribute to the problem (Chute & Wiener, 1994). 


1. Historical background—origins of the two jobs and the influence on 
today’s personal attributes and attitudes. 


2. Physical separation—a serious lack of awareness of each other’s duties, 
responsibilities and problems influenced by a lack of physical proximity. 
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3. Psychological isolation—personality differences, misunderstanding of 
motivations. 


4. Regulatory factors—confusion over procedures and licensing issues. 


5. Organizational factors—administrative segregation, differences in training and 
scheduling. 


The final issue that required a change in the way we train airline crews was identified in 
the early 1990s. Although several airlines had been training a CRM model by this time, 
there was not a large improvement in the performance on the line in measurable outcomes. 
It was discovered that although Line Oriented Flight Training (LOFT) had been initiated 
by several different airlines to promote team concepts there was direct competition with 
the evaluation of flight crews. This competition was caused by the fact that the evaluation 
was conducted under regulations that required the demonstration of several critical 
maneuvers for certification or recertification. Thus, although team and CRM skills were 
identified as critical, they were lost in the technical proficiency required for the check ride. 
(This is what air carrier pilots term bet your job day!) This led to a dichotomy in training 
as compared to evaluation. Although the importance of team skills was understood, the 
focus of the evaluation was completely based on maneuver performance. This resulted in 
the LOFT being shortened in many programs so the focus could be on maneuver 
performance. In the early 1990s the Federal Aviation Administration (FAA) created a new 
training and evaluation methodology known as the Advanced Qualification Program 
(AQP). This program allowed the phased validation of crewmembers in a program 
validated by data to allow a new component of LOFT, the Line Operational Evaluation 
(LOE). The LOE became the new process for licensing a pilot to fly commercial airliners 
in the USA. A key component of the LOE is the assessment of team skills along with 
technical proficiency in simulation of a flight from takeoff to landing. This significantly 
deviated from the old system that required pilots to be evaluated on several maneuvers 
performed in a very artificial environment where there was no time for team. 

This chapter will tell the story of the development of LOFT and LOE in the airline 
industry and the use of this training tool to train, reinforce and assess the CRM skills of 
airline pilots. 


8.1. The Beginning 


The use of full mission or line operations simulation in pilot training programs is 
a technique that has evolved over many years. When the state of simulation technology 
had developed to the point at which the systems operations and handling qualities of 
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a particular simulator were recognizably like those of a specific airplane, when various 
ground facilities, including navaids and airports could be simulated, and when the 
development of visual-scene-generation-technology allowed the simulation of visually 
referenced operations, all the necessary ingredients existed for conducting line- 
operations simulation. 

This technology made possible the use of simulation to teach not only systems 
knowledge, operating skills and aircraft-handling skills, but also the crew coordination, 
decision-making, leadership and management skills, which are important elements of 
the airline pilot’s job. Exigencies and contingencies encountered during “routine” line 
operations could now be simulated, and, under the controlled, safe situation provided by 
the simulation environment, pilots could exercise these “high-level” skills in ways that 
previously could be accomplished only in actual line operations. 

There have been several approaches to the augmentation of simulator training 
programs via the use of line-operations simulation, including a program conducted by the 
United States Air Force Strategic Air Command. However, the most concerted effort, 
which led to a change in the Federal Aviation Regulations (FARs), occurred during 
1974-75. In mid 1974, Northwest Orient Airlines had a task force at work on a program 
known internally as Coordinated Crew Training (CCT). Recognizing that CCT met 
certain training objectives that were not being effectively achieved by recurrent training 
programs conducted under FAR 121 Appendix F, Northwest petitioned the FAA for an 
exemption to permit a one-year test and evaluation of this training concept. The 
exemption was granted in February 1976. On the basis of the positive results observed at 
Northwest, the FAA issued an additional exemption in October 1977, which allowed 
other air carriers to utilize LOFT on a voluntary basis. Finally, in May 1978, Advisory 
Circular AC 120-35 (FAA, 1991) was published, and FAR 121 was amended to permit 
LOFT to be utilized in any airline recurrent training program. 

Since that time, several airlines have implemented LOFT as embodied in AC 120-35. 
Others have evaluated the concept and have taken steps to implement LOFT programs 
in the near future. Still others have evaluated the concept and, for various reasons, have 
decided that LOFT, as defined in the advisory circular, does not meet their 
requirements. Problems cited by these carriers include scheduling, instructor number 
and qualifications, economic costs and, in some cases, concern with the effectiveness of 
LOFT as a method of recurrent training, particularly manual skills training. For a good 
summary of these concerns, the reader is referred to the “Remarks” by Captain A.A. 
Frink which are reprinted in NASA-CP-2184-VOL-2, page 103, Guidelines for line- 
oriented flight training, volume 2 (1981). One of the major objectives of this workshop 
was to review those issues and to develop flexible guidelines which would enable 
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any operator to utilize LOFT to meet his unique requirements (NASA Technical 
Report, 1981). 


8.2. Discussion of Relevant Research 


NASA involvement and interest in LOFT stems largely from some early work 
conducted under a Human Factors in Aviation Safety Program. One major study 
conducted under that program was done by Ruffell Smith and colleagues. This 
study utilized an airline training simulator and highly structured trip scenarios as 
a means of examining human error in flight operations. Ruffell Smith and his 
co-workers were interested in measuring the frequency and kinds of errors in 
simulated line operations and determining the circumstances under which these 
errors were committed. 

One of the earliest observations made during that study was that there may be 
considerable potential for augmenting air carrier pilot-training programs through the 
use of this full-mission, or line-operations, simulation. Specifically, it was observed that 
line-operations simulation seemed to provide a vehicle for demonstrating the 
importance of effective cockpit resource management, and it provided crews with vivid 
demonstrations of operational complications that can result when resources are 
ineffectively or inappropriately utilized. 

These preliminary observations and conclusions were further strengthened when 
the NASA researchers learned of the work being conducted by Northwest. Cockpit 
resource management training was the subject of a NASA/Industry workshop in June 
1979 (Cooper et al., 1980; Lauber, 1981). On the basis of the earlier work at 
Northwest and NASA, and on the basis of the experience with LOFT as described by 
Eastern Airlines at that conference, it was recognized that LOFT provided an 
important tool for conducting cockpit resource management, leadership and 
command training. 


8.3. Definition and Description of LOFT 


With any new or developing technology, problems with nomenclature and the 

definition of terms can arise. Selection of appropriate terminology and definitions is 
an important process—discussions can become hopelessly confusing if terms are used 
imprecisely or if they are poorly or inappropriately defined. Although the problem of 
terminology was discussed at some length at the workshop, and several proposals were 
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made with regard to definitions, terms and acronyms, no consensus was reached. 
Further attention to the problem has resulted in the development of the following 
definitions: 


1. Line Oriented Flight Training (LOFT): refers to the use of a training simulator 
and a highly structured script or scenario to simulate the total line operational 
environment for the purposes of training flight crews. Such training can include 
initial training, transition training, upgrade training, recurrent training and 
special training, e.g. route or airport qualification training. The appropriate 
term should appear as a prefix with LOFT, e.g. “Recurrent LOFT,” to reflect 
the specific application. 


2. Line Operational Simulation (LOS) is synonymous with the term “full-mission 
simulation,” but LOS avoids the other misleading and irrelevant connotations of 
“mission.” LOFT, then, is the use of LOS for training purposes. Any other use of 
LOS should be expressly stated. For example, LOS can be used to aid in the 
development and evaluation of operating procedures and new equipment, 
proficiency checking, pilot selection for new-hire programs, or cockpit human 
factors research. 


8.4. Essential Features of LOFT 


The following quotations from this discussion during the first day of the workshop 
reflect many of the characteristics of LOFT that distinguish it from other forms of 
simulator training: 


LOFT is a line environment flight-training program with total crew participation in real- 
world incident experiences, with a major thrust toward resource management. 
(Captain H.T. Nunn) 


...line-oriented flight training, in principle, has filled a long existing need in airline-crew 
training, that of command and resource management in the total crew resolution of realistic 
line-type problems. 


(Captain A.A. Frink) 


The features that characterize LOFT are as follows: 


1. LOFT is the application of line-operations simulation to pilot-training 
programs. LOFT is a combination of high-fidelity aircraft simulation and high- 
fidelity line-operations simulation. 
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2. LOFT involves a complete crew, each member of which operates as an indi- 
vidual and as a member of a team just as he does during line operations. 


3. LOFT involves simulated real-world incidents unfolding in real time. Similarly, 
the consequences of crew decisions and actions during a LOFT scenario will 
accrue and impact the remainder of the trip in a realistic manner. 


4. LOFT is casebook training. Some problems have no single, acceptable solution; 
handling them is a matter of judgment. LOFT is training in judgment and 
decision-making. 


5. LOFT requires effective interaction with, and utilization of, all available 
resources; hardware, software, and “‘liveware,” or the human resources. A LOFT 
scenario requires the exercise of resource management skills. 


6. LOFT is training. LOFT is a learning experience in which errors will probably 
be made, not a checking program in which errors are not acceptable. The 
purpose of LOFT is not to induce errors, but cockpit resource management is, 
in part, the management of human error. Effective resource management 
recognizes that under some circumstances, such as ‘night-workload situations, 
human error is likely; steps must be taken to reduce the probability of error. 
However, it is also necessary to maximize the probability that error, when it does 
occur, will be detected and corrected, thereby minimizing the probability of 
adverse impact upon the overall safety of the operation. Just as it is necessary to 
practice landing skills in order to gain and maintain aircraft-handling profi- 
ciency, it is necessary to practice human-error-management skills; the former 
requires a simulator or airplane, and the latter the presence of errors or error- 
inducing situations. 


8.5. Limitations of LOFT 


Although LOFT may fill an important training need, the potential user of LOFT 
must recognize that it is not a panacea for all training problems. LOFT is resource 
management training, but, as pointed out frequently in the proceedings of the 
resource management workshop, one of the absolute prerequisites of effective cockpit 
management is a highly skilled, highly knowledgeable pilot. Proficiency in manual 
control of the aircraft and in the operation of its systems is primary—without it, no 
amount of management, command, or leadership training will produce a safe, 
proficient and effective pilot. Therefore, LOFT can be effective only in the context of 
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a total training program that ensures that basic skill and knowledge requirements are 
met. This total program must be based upon the unique requirements of an airline, 
a fleet of aircraft, a crew, or an individual crewmember. Furthermore, these 
requirements are not static. For these reasons, LOFT must not be viewed as a training 
program, but rather as a tool that can contribute to the overall objectives of such 

a program. LOFT is not a replacement for maneuver-oriented flight training, or 
“batting practice” as it has been called. When both are combined in proportions 
determined by the unique requirements of the carrier, a more effective total training 
program will result. 


8.6. The Change of LOFT to LOS and Line 
Operational Evaluations (LOE) 


Since the early 1980s, as the technology of flight simulators and flight training devices 
advanced, the number of training and evaluation applications has increased. These 
training and evaluation applications are now grouped under the general term of Line 
Operational Simulations (LOS). It has become evident that LOS is the most appropriate 
environment to train and evaluate both technical and CRM skills. Consequently, 

a structured LOS design process is necessary to specify and integrate the required CRM 
and technical skills into LOS scenarios. 

As was stated, in the early 1990s there was a strong dichotomy in how airline crews 
were trained. Although the highest risks for airliner crashes were the human factor and 
team skills, the evaluation of the crew in air carrier training was still focused on the 
technical competencies. What was more complex was that the Federal Air Regulations 
(FARs) had no ability to release airlines from these technical requirements during the 
evaluation. Thus, the check ride was too full of maneuver proficiency to allow 
assessment of team skills. The LOFT had become batting practice for the check ride and 
the CRM skills were lost in this practice. 


8.6.1. Joint Government/Industry Task Force 


On August 27, 1987, FAA Administrator T. Alan McArtor addressed representatives 
from major air carriers and air carrier associations, flight crewmember associations, 
commuter air carriers and regional airline associations, manufacturers and government 
organizations. One of the issues discussed at the meeting focused on flight crewmember 
performance. This meeting led to the creation of the Joint Government-Industry 
Task Force on Flightcrew Performance. On September 10, 1987, the task force met at 
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the Air Transport Association’s (ATA) headquarters to identify and discuss flight 
crewmember performance issues. Working groups in three major areas were formed: 
(1) man/machine interface; (2) flight crewmember training; and (3) operating 
environment. Each working group submitted a report and recommendations to the joint 
task force. On June 8, 1988, the recommendations of the joint task force were presented 
to Administrator McArtor. The major substantive recommendations to the 
Administrator from the flight crewmember training group were the following: 


1. Require Part 135 commuters (whose airplane operations require two pilots) to 
comply with Part 121 training, evaluation, qualification and recordkeeping 
requirements (14 CFR Part 119). 


2. Provide for an SFAR and AC to permit development of innovative training 
programs (SFAR 58). 


3. Establish an FAA national air carrier training program office that provides 
training program oversight at the national level (Air Carrier Training Branch, 
AFS-210, now Voluntary Safety Programs Branch, AFS-230). 


4. Require seconds-in-command (SIC) to satisfactorily perform their duties 
under the supervision of check airmen during operating experience (Part 121, 
section 121.434(c)(2)). 


5. Require all training to be accomplished through a certificate holder’s training 
program. 


6. Provide for approval of training programs based on course content and 
training aids rather than specified programmed hours (SFAR 58). 


7. Require Cockpit Resource Management (section 121.404) (SFAR 58) 
training and encourage greater use of LOFT (section 121.409) (SFAR 58). 


8.6.2. National Transportation Safety Board (NTSB) 


In June of 1988, the NTSB issued a Safety Recommendation (A-88-71) on the subject 
of CRM training. The recommendation was that all Part 121 carriers review initial 
and recurrent flight crew training programs. The purpose of this review was to ensure 
that the training programs include simulator or aircraft training exercises that involve 
cockpit resource management and active coordination of all crewmember trainees, and 
which permit evaluation of crew performance and adherence to those crew 
coordination procedures. 
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In response to the recommendations from the joint task force and from the NTSB, 
the FAA, on October 2, 1990, published SFAR 58 (FAA, 1990), which addresses the 
majority of the above recommendations. The Advanced Qualification Program (AQP) 
was also established to permit a greater degree of regulatory flexibility in the approval of 
innovative pilot training programs. Based on a documented analysis of operational 
requirements, a certificate-holder under AQP may propose to depart from traditional 
practices with respect to what, how, when and where training and testing are conducted. 
This is subject to FAA approval of the specific content of each proposed program. Part 
121 subpart Y requires that all departures from traditional regulatory requirements be 
documented and based upon an approved continuing data collection process sufficient 
to establish at least an equivalent level of safety. AQP provides a systematic basis for 
matching technology to training requirements and for approving a training program 
with content based on relevance to operational performance. An applicant may propose 
to replace certain requirements of 14 CFR Parts 61, 63, 65, 121, or 135, with an AQP 
curriculum, subject to FAA approval. An AQP may also employ substitutes for the 
practical test requirements. Each requirement of Parts 61, 63, 65, 121, 135, or the 
practical test standards (PTS) that is not specifically addressed in an approved AQP 
curriculum continues to apply to the certificate holder. The basis of the program is as 
follows. 


Goals 


The overall goals of the Advanced Qualification Program are: (1) to increase aviation 
safety through improved training and evaluation and (2) to be responsive to changes in 
aircraft technology, operations and training methodologies. 


Distinguishing features 


In general an AQP differs from traditional regulatory requirements in terms of the 
following characteristics: 


1. Participation is voluntary. Air carriers choosing not to participate will 
continue to be governed by the appropriate existing provisions of 14 CFR, 
Parts 121 and 135. However, nearly all major US airlines are presently 
participants. A growing number of regional airlines also participate in the 
program. 


2. An AQP may employ innovative training and qualification concepts. Provided 
the applicant can demonstrate to the FAA’s satisfaction that the resulting aircrew 
proficiency will meet or exceed that obtainable through a traditional program, 
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significant departures from the pertinent requirements of FAR Parts 121 and 
135 may be authorized. 


3. An AQP entails proficiency-based qualification. That is, provided that pilots 
are trained to a standard of proficiency on all objectives within an approved 
AQP curriculum, it is not necessary to verify proficiency by virtue of 
a formal proficiency check on every such item. Rather, the proficiency 
evaluation may consist of a sample of such items, in order to validate that the 
training to proficiency strategy has in fact achieved its objectives. Terminal 
proficiency objectives (TPOs), together with associated performance stan- 
dards, replace the FAA’s traditional event-driven compliance requirements. 
Each air carrier applicant, rather than the FAA, develops its own TPOs on 
the basis of an instructional systems development (ISD) process outlined in 
Advisory Circular 120-54, Advanced Qualification Program (FAA, 1991). 
Once approved by the FAA, these TPOs become regulatory requirements for 
the individual carrier. An AQP provides an approved means for the carrier to 
propose TPO additions, deletions, or changes as needed to maintain a high 
degree of aircrew proficiency tailored to the operator’s line requirements. 


Mandatory requirements 


In order to assure that the increased flexibility inherent in AQP does not come at the 
cost of reduced safety, certain mandatory criteria have been established, among them the 
following. An AQP must: 


1. Be aircraft specific (i.e. accommodate each make, model and series or variant of 
aircraft within any given fleet transitioning to an AQP). 


2. Provide indoctrination, qualification and continuing qualification curriculums 
for every duty position. Indoctrination consists of fleet common knowledge 
items. Qualification consists of fleet specific ground and flight operations 
training for a newly assigned cockpit duty position. Continuing qualification 
consists of fleet-specific ground and flight operations recurrent training for 
presently held duty positions. 


3. Provide training and evaluation, which is conducted to the maximum extent 
possible in a full cockpit crew environment (e.g. captain and first officer). 
Qualification and continuing qualification curricula must include a Line 
Operational Evaluation (LOE), which consists of a full flight scenario system- 
atically designed to target specific technical and CRM skills. 
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4. Integrate training and evaluation of CRM in accordance with the provisions of 
Advisory Circular 120-51 (FAA, 2004), Crew Resource Management Training, 
Advisory Circular 120-54 (FAA, 1991), Advanced Qualification Program, and 
14 CFR Part 121, Subpart Y. The evaluation of CRM proficiency is mandatory, 
and substandard performance on CRM factors must be corrected by additional 
training. In AQP, demonstration of proficiency in maneuver-oriented technical 
skills is a necessary but insufficient condition for pilot qualification. For pass/fail 
purposes, pilots must also demonstrate proficiency in LOEs, which test both 
technical and CRM skills together. 


5. Provide AQP-specific training for instructors and evaluators, together with 
explicit training and evaluation strategies to verify the proficiency and 
standardization of such personnel for crew-oriented, scenario-based training 
and evaluation tasks. 


6. Collect performance proficiency data on students, instructors and evaluators, 
and conduct airline internal analyses of such information for the purpose of 
curriculum refinement and validation. Participants are also required to forward 
certain data to the FAA in digital electronic format on a routine basis. 


7. Integrate the use of advanced flight training equipment, including full flight 
simulators. AQP encourages air carriers to utilize a suite of equipment matched 
on the basis of analysis to the training requirements at any given stage of 
a curriculum. Judicious analysis of these requirements can enable an AQP 
participant to significantly reduce the need for use of a full simulator. 


Thus, the creation of a training/evaluation system to assess team skills and technical 
competencies was born. However, much work was still necessary to develop specific 
team skills integrated with technical competencies that could be fairly assessed. 


8.7. The Creation of the Line Operational 
Evaluation (LOE) 


As a result of these committee recommendations the Air Transport Association (ATA) 
and the FAA created industry working groups to create the working model of the 
components of SFAR 58. There were several subcommittees formed that included: 


Crew Resource Management (CRM) 


Instructional System Design (ISD) 
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Data Management (DM) 


In LOFT and CRM there were critical issues that had to be addressed. In particular it 
was moving CRM from a didactic lecture of knowledge to an integrated performance- 
based assessment where the LOFT is used as the vehicle for this assessment. This LOFT 
vehicle became know as the Line Operational Evaluation (LOE) and was a critical 
component of the AQP philosophy. The core element of the LOE is the assessment of 
the team and the competencies of the individual crewmembers. However, before these 
assessments could occur, flight teams had to be identified, team skills linked to 
performance, assessment metrics needed to be established, and pilots needed to receive 
training. 


8.7.1. What is a “Team”? 


To identify the key features of a team for the LOFT focus group reviewed several often- 
cited definitions (Dyer, 1984; Guzzo & Shea, 1992; Mohrman et al., 1995; Salas et al., 
1992) as well as other relevant literature. The definition we adopted for this discussion 
comprises the following five characteristics: 


1. Teams consist of a minimum of two or more individuals. 


2. Team members are assigned specific roles, perform specific tasks and interact or 
coordinate to achieve a common goal or outcome (Dyer, 1984; Salas et al., 
1992; Morgan et al., 1986). 


3. Teams make decisions (Orasanu & Salas, 1993). 


4. Teams have specialized knowledge and skills (Guzzo & Shea, 1992) and often 
work under conditions of high workload (Orasanu & Salas, 1993; Bowers et al., 
1997). 


5. Teams differ from small groups (Salas et al., 1992) because teams embody the 
coordination that results from task interdependency; that is, teamwork character- 
istically requires team members to adjust to one another, either sequentially or 
simultaneously, to achieve team goals (Morgan et al., 1986). 


Examples of teams that fit this definition include military command-and-control 
teams, cockpit crews, SWAT teams, fire rescue teams and management teams. Defining 
what constitutes a “team” is a preliminary step towards identifying measurable variables 
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that reflect team inputs, team processes and team outcomes. In turn, specifying these 
team-input, team-process and team-outcome variables yields a framework that guides 
the design of a given training program and the manner in which the program’s 
effectiveness will be assessed. 


8.7.2. The Nature of Effective Teamwork 


Teamwork has traditionally been described in terms of classical systems theory, which 
posits that team inputs, team processes and team outputs are arrayed over time. In 
particular, team inputs include the characteristics of the task to be performed, the 
elements of the context in which work occurs and the attitudes team members bring to 
a team situation. Team process constitutes the interaction and coordination that is 
required among team members if the team is to achieve its specific goals. Team outputs 
consist of the products that result from team performance (Hackman, 1987; Ilgen, 1999; 
McGrath, 1984). Thus, teamwork per se occurs in the process phase, during which team 
members interact and work together to produce team outputs. Finally, teamwork does 
not require team members to work together permanently; it is sustained by a shared set 
of teamwork skills, not by permanent assignments that carry over from day-to-day 
(Morey et al., 2002). However, simply installing a team structure in an organization does 
not automatically result in effective teamwork. Effective team performance requires 
team members’ willingness to cooperate for a shared goal. Moreover, effective teamwork 
depends on effective within-team communication and adequate organizational 
resources and support. In short, teamwork requires team members to develop a shared 
awareness of one another’s roles and abilities. Without this awareness, serious but 
avoidable adverse outcomes may result from a series of seemingly trivial errors that 
effective teamwork would have prevented. 

Extensive research has yielded numerous models of effective teamwork (Campion 
et al., 1993; Hambrick et al., 1996; Stevens & Campion, 1994; Fleishman & Zaccaro, 
1992; West & Anderson, 1996). Historically, this literature has sought to identify generic 
teamwork skills that are associated with most teams. However, the focus has more 
recently shifted towards identifying the specific competency requirements that team 
members exhibit (Guzzo & Shea, 1992; Stevens & Campion, 1994; O’Neil et al., 1997). 
Although the term competency signifies a variety of meanings, it is generally used to 
denote the qualities needed by a jobholder (Parry, 1998). Specifically, Parry defined the 
term “competencies” as a cluster of related knowledge, skills and attitudes that (1) affects 
a major part of one’s job (i.e. one or more key roles or Boyatzis (1982), in his seminal 
work on competencies, defines a job competency as “an underlying characteristic of 
a person, which results in effective or superior performance in a job”); (2) correlates 
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with successful job performance; (3) can be measured against well-accepted standards; 
and (4) can be improved through training and development (O’Neil et al., 1997). 
Generally speaking, team competencies are the attributes team members need to 
engage successfully in teamwork: As has been suggested, ““...It is essential to understand 
the nature of competencies required to function in a team as a means to define selection 
criteria, design and conduct training, and assess team performance” (Guzzo & Shea, 
1992). To explicate this understanding, Cannon-Bowers and colleagues identified three 
types of competencies that are critical for effective teamwork: (1) teamwork-related 
knowledge, (2) teamwork-related skills and (3) teamwork-related attitudes. 


8.7.3. Teamwork-Related Knowledge 


Team knowledge competencies are defined as the principles and concepts that 
underlie a team’s effective task performance. Broadly speaking, the competencies 
denote that, to function effectively in a team, team members should know what team 
skills are required, when particular team behaviors are appropriate, and how to 
manifest these skills and behaviors in a team setting. Further, team members should 
know the team’s mission and goals and be aware of one another’s roles and 
responsibilities in achieving them. Such knowledge enables team members to form 
appropriate strategies for interaction and to coordinate with their teammates, thereby 
effecting successful team performance. 


8.7.4. Teamwork-Related Skills 


Team skill competencies, defined as a learned capacity to interact with other team 
members at some minimal proficiency, have received considerable research attention 
(Guzzo & Shea, 1992). However, the literature on team skills is confusing, contradictory 
and plagued with inconsistencies in terms of skill labels and definitions. Across studies, 
different labels are used to refer to the same teamwork skills, and the same labels are used 
to refer to different skills. For example, in an attempt to resolve earlier inconsistencies, 
studies have found that 130 skill labels could be sorted into eight major categories: 
adaptability, situation awareness, performance monitoring/feedback, leadership, 
interpersonal relations, coordination, communication and decision-making. Previous 
investigations have shown these skills to be directly related to effective team performance 
(Morgan et al., 1986; Oser et al., 1992; Salas et al., 1999). Nevertheless, a number 

of investigations have demonstrated the difficulty of measuring more than four 
distinct skill competencies during scenario-based training (Brannick et al., 1993a, 
1993b; Smith-Jentsch et al., 1998a). In light of this finding, the best skills to include in 
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an assessment are those that are not only crucial, but also trainable and measurable. One 
research study that exemplified this principle identified four teamwork skill 
competencies related to the performance of air traffic control (ATC) 
teams—information exchange, supporting behavior, team feedback skill and flexibility 
(Smith-Jentsch et al., 1998b). A subsequent study by the same research group reliably 
and accurately measured these competencies during Navy combat-information-center 
team-training scenarios (Smith-Jentsch et al., 1998a). 


8.7.5. Teamwork-Related Attitudes 


Team attitude competencies are defined as internal states that influence a team member’s 
choices or decisions to act in a particular way (Guzzo & Shea, 1992; Smith-Jentsch et al., 
1998b). Attitudes toward teamwork can exert a significant effect on the application of 
teamwork skills. Positive attitudes toward teamwork and mutual trust among team 
members are critical to successful team process (Gregorich et al., 1990; Helmreich et al., 
1986; Ruffell Smith, 1979). For example, Vaziri and colleagues found that higher levels 
of mutual trust among team members led to a more harmonious and productive team 
environment (Vaziri et al., 1988). A later study reported a difference between 
individually oriented individuals, who tend to believe that success is more a function of 
competition than of cooperation, and collectively oriented individuals, who tend to 
endorse the opposite view. In this study, collectively oriented individuals performed 
significantly better than did individually oriented team members because the collectively 
oriented individuals took advantage of the benefits offered by teamwork (Driskell & 
Salas, 1992). Furthermore, the collectively oriented individuals were able to consider 
other team members’ behavior and believed that a team approach was superior to an 
individual one. Thus, as Eby and Dobbins suggest, an attraction to being part of a team 
(i.e. a collective orientation) is a desirable aspect of a positive team attitude (Eby and 
Dobbins, 1997). 


8.7.6. Contextual Factors 


Finally, effective teams do not function in a vacuum. Tannenbaum and colleagues have 
proposed an integrative model of team effectiveness that identifies individual 
characteristics (i.e. ability, motivation) and team characteristics (i.e. power distribution, 
cohesiveness) that are relevant to successful team performance (Tannenbaum et al., 
1992). However, this model also emphasizes the importance of organizational 
characteristics, such as reward systems and organizational climate; task characteristics, 
such as task type; and work structure characteristics, like team norms. 
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Figure 8.2 The Event Set Design Process. An event set designed around this 
scenario incorporates multiple triggers, distracters and supporting events. 
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The (AQP) philosophy allows for the integration of team CRM skills into the 
training and evaluation of flight crews. (Figure 8.2) 


8.8. AQP Program Method 


The excellent safety record of airline aviation that is attributed to the industry’s “culture 
of safety” has emerged over the last 30 years (Sexton et al., 2000; GMU, 1995-97; ATA, 
1994 (Hamman et al., 1993); Hamman, 1994). This state-of-the art airline crew 
proficiency is conducted on the AQP template. Three consistent and intertwining 
trends can be identified that accompany this development of characteristics that, in sum, 
constitute the “safety culture”: 


1. Reporting systems collect real-world system performance data in finer and finer 
detail; these data are shared across the industry. 


2. Standard Operating Procedures—best practices—are developed based on the 
knowledge extracted from these “lessons learned.” 


3. Increasing attention is given to understanding and integrating team performance 
into daily operations and overall professional requirements. 


These team skills were identified by modeling using simulation scenarios employing 
the event set design process. Specific skills (technical and team) are identified by 
criticality measurement and task analysis methodology. The skills are defined as 
a “toolbox” of processes, strategies and tactics empirically identified and characterized in 
studies of crew behaviors conducted in simulated flight. Once the analysis of these skills 
is complete, the skills become the focus for evaluating proficiency and measuring system 
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reliability. In turn, training and reinforcement/remediation using simulation is 
distributed throughout the team member’s career. 

High-fidelity training simulations do not automatically yield effective training. 
Simulations, at their best, reproduce realistic tasks and afford trainees practice that 
enhances learning (Cannon-Bowers & Salas, 1997). In a team setting, simulators allow 
teams the opportunity to practice both team- and task-related skills. Context-specific 
cues embedded within the simulation provide trainees with signals that activate trained 
behaviors. In addition, simulators provide opportunities for team members to receive 
feedback on the actions, activities or strategies performed or not performed. A 
supplementary benefit of simulation training is that it allows training instructors to 
identify performance decrements and particular situations that require further training. 
Under AQP the primary unit of both simulator scenario model design and assessment is 
the scenario event set, a group of related events which are part of the scenario that are 
inserted into a simulator session for specific objectives (Hamman, 1994; Cannon- 
Bowers & Salas, 1997; ATA, 1994 (Hamman et al., 1993); Beaubien et al., 1999; ATA, 
AQP Subcommittee, 1994). The event set is made up of one or more events including 
an event trigger, distracting events and supporting events. The event trigger is the 
condition (or a condition) which fully activates the sequence. The distracters are 
conditions inserted within the event set time frame that are designed to divert the team’s 
attention from actual or incipient team actions. Finally, supporting events are other 
events taking place within the event set designed to further the training objectives. Thus, 
the event set becomes the focus of elucidation and assessment of team skills. Within an 
identified aviation simulation scenario there will be several event sets with different team 
skills required for effective team performance. This process of design of science-based 
team skills and safety science centered on simulation tools create improved team 
recognition of “threats” and errors permitting early interdiction of “accident-chain”’ 
sequences. When “teams” are trained in this manner artificial barriers to information 
sharing are reduced as the “authority gradient” is flattened. 

The Advanced Qualification Program (AQP) was developed to allow the integration 
of technical skills with human factor skills to evaluate the crew in an operational 
environment created in the simulator. AQP has defined Crew Resource Management 
(CRM) or team training as two specific and different areas of CRM Topics and Skills: 


1. CRM Topics and Skills that concentrate on crewmember attitudes and behavior 
and their impact on flight safety. 


2. CRM Topics and Applied Practical Flight Management Skills and intervention 
tools. 
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These different topics and related skills have different instructional strategies. 
Additionally, different groups or cluster of skills apply to specific issues such as error 
management, proceduralized forms of CRM, or autoflight management. An example of 
a skill cluster would be: 


CRM skill clusters 


Topic Sub-Topic e Active monitoring 


Strategic Proactive preparation e Establishing specific parameters 


Planning and guidelines 
e Communication and confirm agree- 
ment and understanding of plan 


Establish limits e Communicate and confirm agreement 
and understanding of limits 


e Define personal and operational limits 


e Alternate plan is initiated 
when limits are exceeded 
Contingency planning e Anticipating outcomes 


e Establish alternatives 


Technical knowledge and e Includes operational limitations and 
experience considerations in plan 

Communicate and 

confirm agreement and 

understanding of plan 


An analysis of human factors in aviation will naturally center on the behavior of the 
aircrew and the operational particulars of any one flight. The primary unit of both 
simulator scenario model design and assessment is the scenario event set, a group of 
related events which are part of the scenario and are inserted into a simulator session for 
specific objectives (ATA, 1994 (Hamman et al., 1993); Beaubien et al., 1999; ATA, AQP 
Subcommittee, 1994). The scenario event set is a refinement of the AQP concept of 
event (FAA, 1991), and like that concept is an integral part of the assessment. The event 
set is made up of one or more events including an event trigger, distracters and 
supporting events. The event trigger is the condition or conditions under which the 
event is fully activated. The distracters are conditions inserted within the event set time 
frame that are designed to divert the team’s attention from other events that are occurring 
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Figure 8.3 The scoring strategy is used to assess the team performance focused 
specifically on event sets designed to challenge flight teams in the LOE 





or that are about to occur. Finally, supporting events are other events taking place within 
the event set designed to further the training objectives, both team and technical 
objectives. The following five steps summarize the simulator scenario design process 
(Figure 8.3): 


The first step is to identify the primary performance categories and integrate them 
with the primary technical training objectives. 


The second step is to apply the performance categories to the design models from 
step 2. 


The third step is to specify simulator scenario objectives, related Terminal Profi- 
ciency Objectives (TPOs), primary and secondary team categories, roles and 
observable team behaviors for each scenario event set. 


The fourth step is to represent the simulator scenario showing the event sets, event 
trigger and the related performance categories, using at least two different 
expert teams, to administer the same validation instrument to expert teams that 
perform the scenario, and make required modifications to the simulator scenario 
to create standard levels of performance. 


The fifth step is to develop final representation of the models in simulator 
scenario with emphasis on event sets and implementation for the performance 
assessment (Figure 8.3). 


The final dimension to the above process is the assignment of success criteria to the 
event sets as developed above. This criterion is critical because it will be used for the 
objectives of the event set, the evaluation of performance in the assessment of simulation 
effectiveness (Figure 8.4). Additionally, this information will be used to identify the 
critical pilot team skills for successful performance in the operational environment. 

The event set team assessment should be viewed as an expansion of the assessment of 
technical proficiency of a specific procedure. Under this system, when a procedure is 
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Figure 8.4 Linked team and technical skills to the overall performance in the event 
set 






Technical 
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undertaken, the team is expected to perform within a given set of performance criteria. 
These criteria are normally based on the outcome of the procedure. Likewise in the 
event set the assessment should be based on the outcome of each event set in the 
simulator scenario. The final assessment for the entire simulator scenario is based on the 
summation of the individual and team performance of the individual event sets 
(Figure 8.5). 

AQP relies heavily on the collection of detailed performance measures to determine 
what aspects of the training are working well and what aspects need improvement. One 
of the hallmarks of the AQP process is that it is a data-driven approach to training. 

In exchange for providing airlines with some degree of latitude to deviate from the 
basic training requirements, AQP-participating carriers must demonstrate—with data 
collected from their own pilot crews—how well their training is working as intended. 
Therefore, the assessment of flight crews has two functions. Number one, it assesses the 
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Figure 8.5 Core assessment of LOEs: the event set 
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program and how well the skills have been taught and number two, it assesses the crew in 
a more realistic environment of flying an airliner during the LOE. The following is an 
example of the integration of a CRM topic into the LOE (Figure 8.6): 

The TOPIC is chosen from the task analysis 


(a) Communications Processes and Decision Behavior. This topic includes 
internal and external influences on interpersonal communications. External factors 
include communication barriers such as rank, age, gender, and organizational culture, 
including inadequate SOPs. Internal factors include speaking skills, listening skills and 
decision-making skills, conflict resolution techniques, and the use of appropriate 
assertiveness and advocacy. The importance of clear and unambiguous communication 
must be stressed in all training activities involving pilots, flight attendants, and aircraft 
dispatchers. The greater one’s concern in flight-related matters, the greater is the need 
for clear communication. More specific subtopics may include the following: 


The SUB-TOPICS are chosen for the Topic 


Inquiry/Advocacy/Assertion. Training in the potential benefits of crewmembers 
advocating the course of action that they feel is best, even though it may involve 
conflict with others. CRM skills under Inquiry may include: 


The CRM SKILLS are chosen for the Sub-Topic 

Actively seeks information. 

Questions are asked in ambiguous or uncomfortable situations until there is clarity. 
OBSERVABLE BEHAVIOR 

An observable behavior may be: Crew questions fuel status. 


These skills would be assigned to event sets in the LOE (Table 8.1). The LOE 
would be trained to the evaluation pilots of the airline to assure inter-rater reliability. 
The airline would evaluate the performance on the LOE to validate the training the 
pilots have gone through, as well as to assure that the pilots are competent in the 
challenges of flying a complex computerized aircraft, in a complex air traffic control 
environment, when complex situations develop. The material for the event sets will 
come from the airlines’ near-miss data reporting and other safety reporting systems. 
The focus of the event sets is to integrate risks identified, which could be the next 
accident, into a training design to provide the pilots with the knowledge and tools to 
mitigate these events from becoming accidents in the real world. Figure 8.6 
demonstrates the integration process. 


Table 8.1 The event set matrix 


Event set 


EVENT SET #1— Pre- 
departure through 
the beginning of the 
takeoff. The crew 
must consider winter 
operations. 


Phase of 
flight 


Pre-departure Push 
back Taxi out 
Takeoff 


Technical 
requirements 


De-icing procedures 
must be followed. 
Takeoff alternative is 
required. 

Takeoff from short 
runway in winter 
conditions with takeoff 
gross weight near 
runway limit. 

Flaps 5/15 takeoff 
required. 

Engine run-up required 
in takeoff position. 


Key events 


Departure, en route 
and arrival in winter 
conditions. 
Destination WX is at 
CAT Illa minimums. 
During pre-flight crew 
may have a duct 


CRM behaviors 


COMMUNICATION: 

Open, interactive crew climate 
established, crew asks questions 
and seeks answers on operational 
issues they are concerned about. 
DECISION-MAKING: 

Captain asks and receives input, 


overheat or wing anti-ice but makes final decisions 


valve fails in position. 
During engine start 
there is no N1 indication 
on engine #1. 

OR 

The #2 engine has a 
hung start, but starts on 
the second attempt or 


when turning the engine 


anti-ice on, one valve 
fails to open. 

Taxi via slippery and 
congested ramps and 
taxiways in low visibility. 


affecting mission. Crew 
continually assesses changing 


conditions to improve operations. 


WORKLOAD MANAGEMENT: 
Efficient workload distribution 
so no one is overtaxed. Uses all 
available resources for complex 
departure. 





EVENT SET #2—Takeoff Takeoff 


through level 
off at 5,000 ft 


Climb 


Engine run-up required 
in takeoff position. 


Cycle gear after takeoff. 


The takeoff runway 
limited, low visibility 
and icing conditions 
near runway limit. 
There is rapidly rising 
terrain to the south of 
the departure runway. 
Complex departure in 
icing conditions. 


COMMUNICATION: 

ATC interaction, problem 
definition about rising terrain. 
WORKLOAD MANAGEMENT: 
Prioritize tasks for departure. 
DECISION-MAKING: 

Captain decisive about rising 
terrain issues, with crew input. 
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Figure 8.6 Integration of CRM skills into the Line Operational Evaluation. The inter- 
section of Team and Technical Competencies 
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8.9. Learning from LOFT/LOE 


Along with these new systems for training came new requirements for the instructor/ 
evaluator in an airline-training program. Traditionally, in technical skill training 
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simulations, the learning occurs during the simulation by repetitive rehearsal of the 
particular task after instructor feedback. In the LOFT/LOE the learning occurs by 
viewing the videotape of the crew performance during the debriefing. During the 
simulation, the team’s performance is captured using sophisticated audio/visual 
recording systems to record examples of particularly effective and ineffective team 
performance. These recordings are then used to help diagnose breakdowns in team 
performance and system safety during a facilitative debriefing that occurs immediately 
following the simulation. The purpose of the debriefing is to help understand the 
complex team skills and knowledge required in today’s world of managing the aircraft. In 
this perspective, the focus is on how behavior impacts safety (both positively and 
negatively) rather than assessing individual performance. The emphasis is on “what is 
right” not “who is right.” To be most effective, rather than teaching or telling the crew, 
it should be a facilitative Socratic debriefing of their performance. This is a special skill 
set that must be trained to instructor evaluators for maximum effectiveness and there 
must be time allocated to conduct this debriefing after the LOFT or LOE training. 


8.10. Summary 


There has been an explosion in technology, capabilities and opportunities for the 
airline pilots in the past 20 years. There have also been frustrations, dramatic changes 
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to the profession and continual threats to our existence. The study and 
implementation of the science of human factors and CRM has done a tremendous 
job in mitigating the challenges faced by crews flying millions of passengers millions 
of miles every year. An integral tool in this process has been the growth of the 
LOFT/LOE to meet the learning needs of today’s pilots. This growth will continue 
with the challenges of training, very small jets, advanced automation in the flight 
deck and changes in how we will navigate the skies in the future. This work will 
continue with the help of industry, research, government and unions. There have 
been countless hours spent by these groups, to reach the ultimate goal of zero 
accidents caused by human performance. Flight safety is as much a human 
performance topic as it is a design topic. Success will come from interdisciplinary 
efforts using methods such as LOFT/LOE to understand and enhance human 
expertise in the context of diverse and dynamic flight settings. 
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Introduction 


On February 12, 2009, Colgan Air flight 3407 crashed into a residential neighborhood 
in Buffalo, New York, killing all passengers aboard and one person on the ground. By 
any measure of flight experience, the crew involved was new to their respective flight 
roles. Given this, the accident is no less tragic, but it is logical that difficult flight 
conditions might hinder an inexperienced crew. On June 1, 1999, American Airlines 
flight 1420 touched down in Little Rock, Arkansas, fast and long on the runway. This 
subsequently resulted in the aircraft leaving the paved landing surface, crashing into 
nearby approach lights. The crash and resulting fire caused 11 casualties including the 
captain, and numerous additional injuries. In contrast to the Colgan tragedy, the crew 
involved in this accident included one of the most senior pilots at the airline’s Chicago 
O’Hare hub (Dismukes et al., 2007). The circumstances leading up to both of these 
catastrophic outcomes were very different, but based on National Transportation Safety 
Board (NTSB) investigations they are bound by a common thread, insufficient training. 

To categorize one of the mitigating factors in these accidents as insufficient training 
could be misleading, however. The knee jerk reaction to these reports is to add a 
component to training that specifically addresses the problem. For example, in response 
to the Little Rock accident, a specific flight training module focused on reverse thrust 
settings in slippery runway conditions might be the resulting recommendation. This, 
however, does not account for the myriad of other, very specific, aircraft settings that 
differ in conditions that may also be inadequately addressed in training. In a perfect 
world there would be unlimited time and money to dedicate to providing pilots with the 
most comprehensive training that exposes each trainee to every known pilot error 
imaginable. However, in reality, training time is limited. In practice, the solution seems 
simple; improve pilot training and assessment—problem solved. Unfortunately, given 
the complexity of aircraft operation, the solution is more complex. 

Current training methods, such as Crew Resource Management (CRM) training, in 
combination with some more traditional methods have been found to produce 
exceedingly positive training outcomes for pilots. The problem with these training 
methods is not that they do not cover every maneuver in flight, no training program can. 
Instead, the issue is the complexity of designing such a training program, which makes it 
difficult for airlines to keep current reliable training programs going. This is not a 
commentary on either airline mentioned in the above examples, but instead, these 
examples serve to emphasize the importance of developing a comprehensive training 
program that continues throughout a pilot’s career. 

The objectives of flight training can be separated into two general functions. These are 
(1) to provide sufficient skill to successfully execute all phases of flight and (2) to prepare 
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pilots to adapt to the dynamic and sometimes unexpected nature of flight. Although this is 
a simplistic breakdown of what comprises aviation training, it encompasses everything 
from the technical skill required to operate the aircraft to the interpersonal CRM skills 
that improve team communication, leadership and decision-making. Line Operations 
Simulations (LOS) provide the most effective, currently available, platform for this in 
aviation training. 

Due to the fact that LOS development is a challenging task for airline training 
departments, we will focus on areas where the process can be made more efficient. In the 
following chapter we will discuss the process of LOS development and illustrate how 
specific tools can be used to supplement the development process. Before discussing 
development, we will first provide a brief introduction on flight training, and more 
specifically, discuss the use of LOS to accomplish flight training goals. 


9.1. Flight Training 


Although pilots are heavily trained on both normal and abnormal flight conditions in 
initial training, throughout a pilot’s career flight experience will vary. These experiences 
will shape how an individual reacts in certain situations. In some cases, this may result in 
the development of behaviors that do not work in all flight conditions. This could be 
related to weather conditions, aircraft functionality, or even occurrences in the cabin. 
For example, the highly experienced flight crew, in the Little Rock accident discussed 
earlier, faced a combination of several conditions resulting in a fast, long landing onto 
a slippery runway. Improper reverse thrust settings led to loss of control and subsequent 
crash. The NTSB suggested that despite the flight crew’s extensive experience, they may 
not have received adequate training on reverse thrust on a wet landing surface (Dismukes 
et al., 2007). This helps illustrate the importance of varied and continuous training 
throughout a pilot’s career. 

A majority of the flights that a pilot will experience over their career fall into the 
category of routine flight, where practiced procedure will be sufficient for safe operations. 
This, unfortunately, does not adequately prepare a pilot to react to ambiguous and 
dynamic situations which infrequently arise in the cockpit. It is impractical to try to train 
individuals to learn every possible known combination of events that could occur due to 
the sheer volume of training that would be required. In most cases, in less than a year’s 
time, training decrement begins to occur especially for rarely used flight skills (Arthur 
et al., 1998; Childs & Spears, 1986). This is not to mention the propensity for new, 
previously unexperienced (or unreported), events to occur. In addition to varying flight 
experience, technological advances and organizational shifts will result in changes over 
the course of a pilot’s career which also will require training. As a result, aviation requires 
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continual training to maintain operational know-how in less frequently occurring 
situations. 

Currently used experience-based training methods have been widely accepted and 
generally found to enhance previous training techniques. The intention of these 
methodologies is to provide pilots with simulated experience that would closely 
match what is required on the line. The idea is based on research in recognition- 
primed decision-making (Klein, 2008) which suggests that varying experiences drive 
mastery of complex skills. That is, the more experiences from which an individual has 
to draw from memory, the better equipped they are for completion of a task in 
a variety of situations. In many cases, pilots encounter similar flight conditions on 
a daily basis. This helps reinforcement of general flight skill, but does little for 
infrequently occurring events. 


9.1.1. Line Operational Simulations 


Although there is still a substantial class lecture aspect to flight training, the most 
important learning is gained through flight experience. In practice, actually flying an 
aircraft is logically the best way to achieve this. Unfortunately, it is impractical to 
provide extensive in-flight training to pilots. Operation cost, increasingly congested 
airspace and safety are among the primary reasons why in-flight training is limited to 
the final phase of training before pilots join line operations. Because of this, the next 
best option is using computer-generated simulations of flight, which consist primarily 
of simulated flight in high-fidelity motion simulations housed at specific training 
facilities. Since the cost for these types of simulations is still relatively high, there are 
a limited number of available simulators running nearly continuously to accommodate 
the high volume of training sessions needed. As a result, it is imperative that simulator 
sessions provide maximal flight realism and optimized experience. Simply providing 
operators with free flight simulator sessions does not insure exposure to all relevant 
training goals and is subsequently inefficient use of simulator time. Instead, sessions 
should incorporate pre-planned scenarios that build off of what was learned in 
classroom lecture and provide practice executing important operations, in a realistic 
cockpit environment. The most practical form of training currently in use which 
provides a platform for what is described above is accomplished through the 
development of LOS. 

LOS has been used to describe the development of realistic flight scenarios for use 
in any simulator training event (Chidester, 1993). More specifically, LOS refers to a 
number of similar methods of flight training and evaluation that reproduce gate-to-gate 
operations in a simulated environment (FAA, 2004). Unlike part task trainers, the main 
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purpose of LOS-based training is not to provide specific instruction on an individual 
aspect of flight, but instead to provide experience with combining technical and CRM 
skills into line operations. Beyond the logistical aspects, such as technological 
requirements, supplemental material (i.e. pre-flight paperwork, checklists, etc.), and 
training department involvement, the driving force in LOS effectiveness is scenario 
development. 

The scenario- or simulation-based training (SBT) method that LOS utilizes is widely 
used in a number of domains including military, medicine and aviation applications (Salas 
et al., 2008; Salas et al., 2006a). The crux of SBT is in providing a platform for trainees to 
gain experience through realistic task environment simulation. Instead of isolating 
individual skill sets, SBT provides a training context where an individual must integrate 
skills to achieve a realistic simulation of the real world task. This is accomplished through 
the use of embedded events, within a scenario, that are designed to specifically elicit 
certain behaviors (Salas et al., 2006a). Beyond the experiential learning benefits 
associated with SBT, it also provides an effective means for observation and evaluation of 
target skills. 

Primarily, LOS-based programs are focused on one of two things, training or 
evaluation. In the following sections we will briefly describe the main ways that LOS is 
used in an aviation setting. This will provide context for discussion on the critical 
features that must be focused on in the LOS development process. 


9.1.2. Types of LOS 


Under the Federal Aviation Administration (FAA) advanced qualification program 
(AQP) several training and evaluation programs that would be characterized as LOS 
have gained support (Birnbach & Longridge, 1993). Two of them, Line Oriented 
Flight Training (LOFT) and Specific Operational Training (SPOT), are methods of 
training, while a third, Line Operation Evaluation (LOE), is used to assess pilot mastery 
of training objectives. Although each is based on the developmental ideals behind SBT, 
the intended purpose of each differs slightly. 


LOS for training 


Both LOFT and SPOT are methods intended to provide a training platform in which 
individuals can practice technical and CRM skills without fear of negative consequences, 
especially in the event of a failure to complete a flight within acceptable safety parameters. 
LOFT is a full gate-to-gate simulator scenario intended to provide flight experience 
including simulation of all pre-flight, in-flight and post-flight events. It is used both for 
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qualification and recurrent training purposes. This includes everything from completion 
of preparations, paperwork, communication with air traffic control and company 
facilities, and performing routine procedures for a normal flight (Chidester, 1993). LOFT 
training is regarded as one of the most effective forms of incorporating technical and 
CRM skills in training (Helmreich et al., 1991; Jensen, 1989). The inclusion of CRM in 
training has been found to elicit a positive change in attitude and behavior (Helmreich & 
Foushee, 1993), 

SPOT is intended to address more specific training objectives. Although in some 
cases this involves full flight simulation similar to LOFT, more often, SPOT is made up 
of partial flight segments. This provides trainers with the flexibility to attend to areas of 
specific training needs, efficiently. This is especially useful when new technologies are 
introduced in the cockpit. For instance, if a new glideslope indication display is being 
introduced into a cockpit, that technology is really only useful in approach and landing 
phases. SPOT is also useful if a trainee needs remediation on CRM skill in a specific 
phase of flight. Because time is often a constraint for pilots and trainers alike, it is more 
efficient to provide abbreviated flight simulations to address these specific areas. 

Although similar, SPOT and LOFT are not interchangeable training methods 
(Butler, 1993). Outside of the obvious potential differences in length and depth of 
scenarios, training personnel involvement also varies. The role of the instructor in 
LOFT is to take on the role of non-cockpit personnel during the flight, providing radio 
calls or cabin crew interactions throughout the flight. In fact, FAA advisory circular 120- 
35C (FAA, 2004) recommends that instructors avoid interrupting LOFT scenarios to 
provide instruction. By withholding instructional feedback in flight, LOFT scenarios 
can capitalize on the benefits of pilot self-realization through decision-making and crew 
coordination (Helmreich & Foushee, 1993). In contrast, SPOT is more dependent on 
the objectives of the specific training. In some cases this allows for instructors to 
intervene in a scenario and provide feedback. 

Depending on the objective of training, LOFT and SPOT both can provide 
beneficial training outcomes. Whether geared toward a specific training goal or to 
overall flight proficiency training, these methods provide a platform in which 
simulated flight is optimized to most closely mimic flight conditions. By providing 
a no jeopardy exposure to full flight scenarios, these types of training allow for flight 
crews to test out numerous strategies for both technical and CRM skill in the 
cockpit. All in all, both of these methods rely on the development of scenarios that 
give practice in both normal and abnormal flight conditions. In addition to being 
used for training purposes, LOS can also effectively be used as an evaluation tool. In 
the following section, we will briefly describe the LOE before further discussing LOS 
development in general. 
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LOS for performance evaluation 


Another similar application of LOS is through performance evaluation. LOEs assess pilot 
proficiency on targeted skills deemed relevant to overall flight safety. In execution, LOE 
is very similar to LOFT. Instructors do not interfere with the flight crew as they conduct 
a realistic gate-to-gate simulation. Where LOE differs is that the observation and 
assessment of performance can influence pilot career advancement. As well as 
developing scenarios that address specific skills, for LOE valid evaluation of those skills is 
very important. Due to the safety concerns in aviation, there is low tolerance for 
inconsistent measures of performance, to be used to help dictate a pilot’s fate. In light of 
this, additional care must be taken to ensure that assessment is consistent and fair across 
flight crews being evaluated. 


Summary of LOS types 


As illustrated above the LOS is gaining ground as an important training and evaluation 
methodology. LOFT and SPOT are both training implementations that are heavily 
dependent on scenarios that provide flight experience as close to what one would 
experience on the line without being in the aircraft. The goal of these is to provide 
a means of practicing both technical and CRM skills in observable event sets. Similar 
to these two training methods, LOE is the evaluation version of LOS. Instead of 
providing a platform for practice, the LOE is intended to evaluate performance on 
technical and CRM skills. Since the LOE is used to make decisions on whether or not 
a trainee is capable of safely operating an aircraft, it is important to develop consistent 
measures of performance from scenarios. As AQP becomes more widespread among 
airlines, the demand for these types of LOS applications will increase. Although we 
outlined the primary applications of LOS, improving the development process may 
lend LOS to additional applications beneficial for training and evaluation. In the next 
section we will discuss the process of developing scenarios, including discussion on the 
numerous ways that make LOS development challenging without the aid of 
development tools. 


9.2. Developing LOS Scenarios 


Developing LOS scenarios requires a thorough process in which attention to detail is 
crucial to achieving training or evaluation goals. The success of LOS development is 
contingent on construction of scenarios that fulfill these training or evaluation goals. 
Due to the range of technical and CRM skills that occur in various phases of flight, it is 
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not sufficient to develop a general scenario where behaviors might be observed. Instead, 
since the scenario is generally comprised of an entire flight from cockpit entry to cockpit 
exit, simply dictating a handful of malfunctions during the course of the flight does not 
guarantee practice with specific skills. For LOS to be effective, the wide variety of tasks, 
environmental variables and possible interactions in a given flight should be considered. 
In order to accomplish this, LOS developers break the overall scenario into smaller parts 
or events. Event sets are smaller segments of the scenario in which a specific event is 
triggered to elicit target behaviors. The Event-Based Approach to Training (EBAT) is 
intended to initiate opportunities for practice or evaluation of target skill sets (Fowlkes 
et al., 1998). By using event sets, LOS developers can build larger scenarios that target 
multiple skill sets in different phases of flight. 

The LOS development process has been described in a number of ways. The first 
step in this process is identifying the technical and CRM skills that need to be addressed 
and identifying aviation events that occur which can elicit demonstrations of these skills. 
Following the identification step, the information gathered must be formulated into 
logical event sequences and evaluated by subject matter experts for accuracy and 
effectiveness. After this has been accomplished and all required approvals have been 
acquired, instructor preparation documents and LOS materials can be developed. Prior 
to widespread use, the final product LOS should be assessed to ensure the scenario is 
a valid and reliable rendition of the objectives outlined for the LOS. FAA advisory 
circular 120-35C (FAA, 2004) outlines the process of scenario development in five steps. 
For the purposes of this chapter, we have collapsed these steps into three categories of 
action, for discussion. These are (1) identification of training objectives, (2) scenario 
building and (3) assessment. In the following sections we will further describe how each 
is accomplished and the challenges associated with each. 


9.2.1. Objective Identification 


The primary step in LOS scenario development is the identification of the objectives for 
training or evaluation. This step is similar to what would be recommended in any 
training/evaluation development process. Logically, before any event sets are developed, 
one must determine what is important for training or assessment. In aviation, this is an 
especially challenging task. The complexity of flying an aircraft does not lend itself to an 
easy isolation of target objectives. The variation of possible aircraft, phases of flight, 
events, behaviors, environmental conditions and organizational structure are an 
oversimplification of the categories of difference that can occur in aviation. As a result, 
no generic scenarios can be built for all pilots or all airlines. Each circumstance calls for 
variations. 
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The identification of objectives in LOS is driven by the need for event sets to 
realistically populate scenarios (Prince et al., 1993). To do this effectively, scenario 
designers have to identify objective skills to target in the LOS scenario in addition to 
identifying cockpit events that will elicit these skill behaviors. 


Identifying target skills 


The multitudes of complex technical, interpersonal and environmental interactions that 
make up each flight make it difficult to cover all flight scenarios in training. Instead, as 
mentioned earlier in this chapter, training relies on building a solid foundation of skills 
that can be applied in a variety of circumstances. In order to do this successfully, 
designers have to identify both technical and CRM skills that will best equip trainees to 
react in the dynamic environment of the cockpit. Identification of technical skills from 
a designer’s standpoint is a relatively straightforward task. Technical skills involve those 
specific to flying an aircraft. This includes proficiency with tasks that would fall under 
the label of “normal” flight in addition to less frequently occurring tasks such as adverse 
weather or flight plan revision. Until the genesis of CRM, technical skill development 
was the focus of training. Since technical skill is defined by the technology present in the 
cockpit, training objectives can be determined by the procedural steps that are required 
to accomplish a task. Technical skills are driven by the aircraft itself. Although much of 
flying is based on the general principle of lift and drag, the function of the system that is 
used can vary between aircraft and flight route. As a result, identifying the range of 
technical skill that will most adequately address the range of technical demands on the 
pilot is most useful. 

As evidenced by other chapters in this book, the use of CRM training has become 
an important feature in the aviation industry. It has also been utilized successfully in 
other domains such as military and medicine (Helmreich, 2000). Despite the success 
with CRM, there is not a blanket CRM training methodology that can be applied in 
all circumstances. In line with this, it is important to identify the specific CRM skills 
that need to be developed and observed for success in aviation. Unlike technical skills, 
CRM skills do not have as direct an operational counterpart. There have been 
a number of skills that have been identified under the label of CRM. This includes but 
is not limited to such skills as communication, planning, leadership, decision-making, 
assertiveness and adaptability (Salas et al., 2006b). Designers should have more than 
a passing understanding of what each CRM skill involves. That is, simply saying pilots 
should display communication ability is not sufficient to identifying a CRM objective. 
Instead, LOS designers should keep the individual crew member roles in mind when 
selecting the appropriate CRM skills to focus on. Assertiveness, for example, is 
a critical skill in the cockpit. The need to observe or train this skill may differ between 
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crew members. A new hire first officer may not be immediately willing to offer their 
opinions on how to proceed. This would obviously warrant focus in training. On the 
other hand, a captain that has had years of experience with crew interaction both as 
a captain and as a first officer may not need as much attention on how to be assertive, 
but may not know on how to receive assertive behavior from their first officer. After 
all, a captain who does not recognize and accept assertive behavior from their crew 
may put crew and passengers in an unnecessarily dangerous situation if the first officer 
recognizes a threat to safety that is disregarded by the captain. By identifying this as 
a target training objective, an LOS can be developed to address important skill 
development at different phases in a pilot’s career. Unfortunately, the nature of CRM 
skill makes it such that there is no one-stop resource for designers to reference in terms 
of what CRM behaviors are critical. 


Identifying aviation events 


In unison with identifying the target skill objectives of training or evaluation, LOS 
designers must also identify flight events that coincide with these skills. Since a major 
part of LOS is contingent upon the ability to observe technical and CRM behaviors, just 
putting a generic flight scenario together and waiting for the behaviors to surface is not 
an efficient method. Instead, identifying specific events that elicit behaviors is necessary. 
This can be accomplished by identifying the operational equivalent, in the aircraft, to the 
technical skill. For instance, if revision of flight path is one of the target technical 
behaviors, a flight event that would elicit this skill would best be served in a phase of 
flight, where individuals may have difficulty making the necessary flight changes. For 
flight path revision this would be most likely to occur in the approach and landing phases 
of flight. Unfortunately, defining CRM skill objectives for training is not as easily 
categorized as technical skill. Overall, CRM-related skills, such as leadership or 
decision-making, may be easy to identify but more difficult to match with events that 
would trigger the behavior in the cockpit. Automation failure by itself does not 
automatically elicit leadership behavior. Instead, the context in which system failures, 
unexpected weather changes, or other events occur will influence the propensity for 
a CRM skill to occur. Whereas technical skill can sometimes be observed as simple 
when considering whether or not the pilot flipped the appropriate switch, CRM skill 
observation is a much more complex undertaking. 


Summary 


For the objective identification phase of scenario development, LOS designers have 
to be careful not to fall into the trap of relying on the same objectives year after 
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year. As cockpit technology is evolving, the nature of pilot tasking is changing as 
well. This means that objective identification cannot satisfactorily be accomplished 
by going through flight procedure once, then relying on it to guide future scenario 
development. Instead, developers have to stay current with the evolution of the 
aviation industry and allow that to help guide the development of scenarios. As 

a result, a scenario that was developed several years ago may have a focus on aviation 
events that are now outdated. Due to this, the task of objective identification is an 
ongoing process for LOS designers. Individuals should stress the importance of 
monitoring industry changes and current trends that can indicate where there may 
be training needs. 


9.2.2. Scenario Building 


After identifying target skills and aviation events that coincide with them, the next step is 
to combine all of these objectives into full flight scenarios. Scenarios should be 
operationally relevant and believable while testing crew skill (Butler, 1993). Taking the 
training objectives and flight events that coincide with them, designers have to combine 
these parts into a coherent whole. Due to the complexity of an entire flight, and the 
number of individual events that can influence how a flight transpires, scenario building 
can quickly turn into an arduous process. In order to build a scenario effectively, the 
identified training objectives and corresponding events have to be further developed 
into event sets. These event sets then must be organized into the structure of a flight to 
create a realistic full flight mission. 


Combining events 


Events are the basic building block of scenario design which are then compiled into 
event sets. Each event set is comprised of an event trigger, which initiates the action on 
the specific event (Johnston et al., 1997). Additionally, events can contain distracters and 
supporting events to enhance the realism of the event and help promote specific 
technical or CRM behaviors (FAA, 2004). 

Event sets are classified as either simple events or complex events depending on how 
each can be resolved. A simple event, once addressed, requires no further action. Many 
simple events may be addressed by referring to procedural manuals. For example, 

a TCAS warning alert may trigger action of an avoidance maneuver; once this is 

completed, there is no further action required. Conversely, complex events do not have 
a well-defined solution and can have a continuous effect over the duration of the flight. 
The failure of any flight system can have more complex corrective action. Failure of an 
automated system, for example, in the flight deck will result in the crew having to adapt 
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from monitoring the state of the automated system, to manually taking over the function 
that the automation drove. In order to get the most realistic scenarios, a mix of simple 
and complex problems should be included in the overall scenario. Too many of either of 
these can detract from realism though. As a result, designers have to find an appropriate 
balance of event sets, dispersed throughout the flight. 

In addition to balancing simple and complex events, designers also have to find 
a good mix of both proceduralized and non-proceduralized event sets. Proceduralized 
event sets follow specific rules for resolution, which can often be addressed by 
referencing to procedural manuals and generally require very little observable CRM 
behavior to rectify. Non-proceduralized event sets, in contrast, cannot be addressed by 
following an established corrective procedure. Instead, non-proceduralized events 
require the flight crew to use knowledge-based solution management to address the 
problem. In many cases, non-proceduralized events will elicit more CRM behaviors. In 
order to effectively resolve an issue, the flight crew will have to engage in a process that 
may include brainstorming solutions, making the decision of which alternative is the 
best course of action, and then executing the action. By providing a non-proceduralized 
event, success hinges on whether the crew effectively utilizes both technical and CRM 
skill to address a unique flight occurrence. 


Instructor workload 


For training, one misconception is that scenarios should progressively increase in 
workload until the trainee experiences overload (FAA, 2004). This, however, is not the 
case. Instead, scenario development should reflect the normal progression of events in 
a flight. Not only is trainee workload important, but design with instructor workload in 
mind is also critical. If the instructor is not able to keep up with all of the activities in the 
LOS, the gains from feedback and evaluation will be lost (Beaubien et al., 2004). 
Considering this, scenario design should make sure that there is not an overload of target 
skills per event set, that event sets are not too short and that the crew gets multiple 
opportunities to demonstrate the target skills. These will improve the quality of the 
observations being made, and also will help maintain appropriate workload of both 
instructor and crew. 


Summary 


Although it is impossible to predict and account for every possible incident that can 

occur, it is important to provide continuous training on both routine and less frequently 
occurring conditions. In order to effectively accomplish this, a variety of event sets and 
subsequent scenarios have to be developed for use. To do this, designers have to be sure 
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to include enough event sets that address skill needs, while at the same time maintaining 
a realistic flight. Training value may be lost if the pilot does not believe the scenario 
could really occur. As a result, in addition to finding the appropriate training objectives, 
designers also have to be sure that the progression of events makes sense from an 
operational, environmental and common sense point of view. 


9.2.3. Scenario Assessment 


The final and perhaps most important phase of LOS development is the assessment 
phase. Because providing a variety of scenarios is critical to the success of an LOS 
program, it is important that scenarios are generated to produce consistent experiences 
from scenario to scenario. Producing a consistent scenario result is critical to providing 
equal training opportunities, and is even more important when the LOS is used for 
performance evaluation (Dismukes, 1999). Designers have to come up with an effective 
way to assess the scenario for difficulty and operational relevance to the training or 
evaluation goals (Birnbach & Longridge, 1993). The goal is not to produce scenarios 
that are so difficult that failure is likely, but to find an acceptable range of performance 
from which performance evaluation is valid and reliable. 


Assessment of LOS 


Since different scenarios can result in differing CRM and technical skill, difficulty of 
problem resolution and workload imposed, there is no clear cut method of making 
comparisons between each (Chidester, 1993). Developing a rating system seems most 
logical, but LOS is made up of multiple layers of events and objectives. Since raters tend 
to underestimate overall difficulty as they rate smaller parts of the scenario, it is 
challenging to come up with the most accurate measure of performance (Jentsch et al., 
1999), 

LOS designers are challenged to develop scenarios to make sure that they are not too 
broad or too focused. Jentsch et al. (1999) investigated a number of ways in which to 
achieve accurate difficulty ratings of the scenarios. They found that difficulty ratings 
derived by simply adding or averaging task component ratings did not result in accurate 
difficulty scores. Instead, they found that averaging the task difficulty within the phase 
and averaging the phase-based ratings into an overall difficulty rating preserved the 
relative differences in difficulty scores. This gives the best estimate of overall difficulty. 
Since these difficulty scores are component based, they are narrower than those based on 
overall LOE, which suggests that one would have to set cutoff scores for too easy or too 
hard items. They suggested that this method of evaluation makes it tough to find 
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a balance between a narrow band for consistent experience and allowing a wide range of 
combinations of events. 


Assessment of performance 


In addition to assessing LOS scenarios for consistency, scenario designers also have to 
consider how trainees will be evaluated in the process. In any case, LOS effectiveness 
hinges on the ability to provide useful feedback to trainees and a valid measure of 
performance in evaluations. Developing performance assessments for proceduralized 
events can be organized into a checklist of actions that should occur to successfully 
address the event. Developing checklists of action for non-proceduralized events that are 
intended to elicit CRM behavior is not as easily accomplished. If the target behavior is 
communication, simply checking a box if the crew communicates will not be sufficient. 
Instead, developers have to find specific communication behaviors that would signify 
success in the event. 


Summary 


In order to fully realize the potential of LOS training and evaluation, the assessment of 
scenarios, in addition to the assessment of trainee performance, has to be included. 
Development of event sets should be built for easy examination of crew action, especially 
for CRM skills where performance is ill defined. Ultimately, designers have to consider 
that assessment is going to involve a subjective human component, and that there are 
subsequent shortcomings that result. 


9.3. Need for Developmental Tools 


Currently, LOS development is not a flawless process. The aviation industry is largely driven 
by monetary and time cost influences. For LOS development, this presents some challenges. 
The development process, described above, from conception to implementation can be 
a lengthy and involved endeavor, just to develop one scenario. Without the benefit of 
developmental aids, costs can begin to soar to keep up with industry demands. To be 
effective, LOS programs have to be comprised of multiple scenarios. Otherwise a limited 
number of scenarios to use for LOS can result in limitations to the scope of what can be 
trained. For evaluation purposes, a limited number of scenarios also run the risk of 
becoming known to trainees prior to the evaluation. This has serious implications for the 
validity of measuring performance. In order to combat this, it is important to have 

a satisfactory pool of scenarios from which to choose. In order to produce a more efficient 
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scenario development cycle, there are a number of tools have been developed to address 
important needs. In this section we will describe the tools that can be used in each phase of 
development. 


9.3.1. Safety Report Databases 


In order to effectively identify objectives, LOS designers need a solid understanding of 
the types of skills that go into flight, and how and where these skills fit into different 
phases of flight. 

In addition, they need to be aware of new cockpit instrumentation, overall aircraft 
advances and organizational changes to be sure that the objectives they are identifying 
link up with current needs. When identifying training objectives, LOS designers should 
consider frequently misunderstood parts of the flight manual, recently reported flight 
incidents and observed poor performance areas that need specific attention. Although 
direct observation from training personnel about issues that they are observing in 
training and practice can help guide this step of design, by itself this method of objective 
identification could lead to an unnecessary narrowing of the objectives of training. If one 
pilot at an airline is having difficulty remembering to raise the gear in a timely fashion 
after takeoff, although the LOS developers observe this, it would not be practical to 
expand training on gear procedure to accommodate everyone based on this individual. 
In order to get a broader picture of the issues that are occurring, there are additional 
safety reporting tools that can help to more thoroughly inform this process. 

Safety reporting systems are an excellent resource for extracting data of current 
trends in incidents that occur in flight. Industry sponsored programs such as the Aviation 
Safety Reporting System (ASRS) and Aviation Safety Action Program (ASAP) provide 
anonymous safety reporting programs where flight crew members voluntarily report 
safety issues they encounter. Since participation is the critical driver of the success of 
these programs, it is important to understand that this method will not capture all 
incidents that occur. Safety reporting systems do provide a broader representation of 
aviation issues than individual training department personnel observations. Other 
sources of information such as NTSB accident report databases and internal airline 
databases can also be helpful in identifying industry trends. 

Safety reporting tools are beneficial in informing the objective identification aspect 
of LOS design, but should be noted as not being an automated system. That is, safety 
reporting systems are databases of information that can be used to assess current trends, 
but there is no current tool that keeps real time information on the issues occurring. The 
availability of this information still must be met with a critical eye to identify relevant and 
irrelevant trends in the data. 
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9.3.2. RRLOS 


As outlined in the previous sections, the process required to build a scenario even after the 
pieces are assembled is still no small task for developers. Developers must balance realism 
of events with objective goals, and generate useful supporting materials for LOS sessions. 
Provided that each scenario must go through this process, in addition to an approval 
process prior to implementation, the amount of time can become an issue for airline 
training departments. Rapidly Reconfigurable Line Operations Simulation (RRLOS; 
aka RRLOE) is a freely distributed program designed to automate the event set pairing 
process to provide quick, valid LOS scenarios (Bowers et al., 1997; Jentsch et al., 2001). 
RRLOS is a system that generates either full or partial flight LOS scenarios (i.e. LOFT, 
SPOT) from a database of existing event sets. The notion is that ifindividual event sets can 
be approved by the FAA (or other international governing bodies), these events can then 
be combined to form full sets of pre-approved LOS flight scenarios (Hitt et al., 2000). To 
do this, RRLOS takes advantage of algorithmic calculations that provide both useful and 
realistic combinations of event sets. By creating a program that is sensitive to aircraft 
variation, weather patterns and general flight characteristics, RRLOS avoids producing 
random but unrealistic scenarios. RRLOS generates a phase-by-phase log of realistic 
flight events that link up with a scripted series of pre-flight paperwork, air traffic 
clearances and evaluation forms for the trainer (Prince & Jentsch, 2001). 

RRLOS provides the flexibility to generate random scenarios, or specify precise event 
sets. In addition, RRLOS utilizes a difficulty rating system similar to what was described 
previously to aid development of scenarios that do not differ in difficulty, but provide 
variation in the types of events that occur. This method makes it easier for LOE developers 
to maintain consistency between LOS sessions without the risk of information breach 
which can occur if only one scenario is available for use. 

Although RRLOS is an effective means of reducing the amount of time required to 
develop individual LOS scenarios, the program is only useful if it is continually updated 
to account for new challenges that arise in the area of aviation training. Even so, if 
a training department stays current on the trends in aviation, which they should anyway, 
updates to RRLOS would still require significantly less time than if an LOS designer 
produced scenarios without it. 


9.3.3. TARGET 


Since LOS-based methods hinge upon the evaluation and interpretation of performance 
in realistic flight scenarios, it is critical to come up with valid and reliable measures of 
performance. As mentioned above, especially for CRM skills, behavior observation is 
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not always easy to extract from an ongoing flight scenario. Due to this, it is important 
that not only the events within a scenario facilitate observable behaviors, but also that the 
method of evaluating performance accurately reflects the behaviors that are being 
targeted. If scenarios are constructed effectively, there should be observable behavioral 
markers that help trainers to assess performance on less well-defined skill areas (Flin & 
Martin, 2001). 

One method of assessing these is known as Targeted Acceptable Responses to 
Generated Events or Tasks or TARGET. TARGET is a rating system that, if correctly 
developed, provides explicit observable events that evaluators can use to determine skill 
development in non-procedural tasks (Fowlkes et al., 1994, 1992). Following the guidelines 
of TARGET development can provide a useful performance assessment tool. Due to the 
fact that human observers have a historically low inter-rater reliability, it is important to 
provide assessment methods that reduce this discrepancy as much as possible. Carefully 
constructed TARGET assessments have been found to display high inter-rater reliability 
and sensitivity to crew performance (Fowlkes et al., 1994). At its best, the TARGET 
method of assessment reduces the need for highly trained subject matter experts to be 
present for behavior observation (Stout et al., 1997). This means that this assessment tool 
can not only be consistently used by different raters, but also that performance measures 
taken will effectively discriminate between high- and low-performing flight crews. 


9.4. Conclusion 


This chapter was intended to provide a brief discussion on the process of developing 
LOS for aviation training and performance evaluation. Currently, LOS-based methods 
(i.e. LOFT, SPOT and LOE) have been found to be effective alternatives to live flight 
missions for training and evaluation. Unfortunately, the process of developing these has 
so far been an expensive and timely venture. As a result, training departments may settle 
for creating fewer scenarios to save money. Since the continued effectiveness of an LOS- 
based program is contingent upon providing varied and relevant experiences to trainees, 
if a training department is forced to reduce the number of available LOS scenarios to 
draw from, they run the risk of providing inadequately diverse training or evaluation 
programs. This is why the utilization of current development tools and the development 
of additional tools are important to fully realize the benefits of the LOS methodology. 
To say that the accidents outlined at the beginning of this chapter were solely a result 
of poor training design would be inaccurate. The countless factors that can cause an 
accident to occur are too numerous to pinpoint the lack of training as the cause. What 
those examples do provide is an illustration that despite improvements in training 
methodologies, errors can occur at any stage of a pilot’s career and that methods of 
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improving the development process can only serve to benefit pilots, airlines and the 
safety of the general public. 

Given the dramatic restructuring of the current landscape of aviation, through the 
future generation of industry-supported programs, the need for efficient training is 
becoming increasingly important. Taking into account the developmental challenges 
associated with LOS, the only way to ensure that this effective method of training and 
evaluation remains viable is to utilize tools to help offset the monetary and time 
constraints currently in place. 
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Introduction 


This chapter begins with a one paragraph general description of Crew Resource 
Management (CRM), and then defines the term “company operations plan” as the plan 
flight crews’ execute to fly every flight to a safe conclusion. When flight crews use their 
CRM skills to execute their flight plan, it is intended to result in a safe operation. 
Measuring the variance in the safety margin, based on flight crew performance, during 
that flight is what a Line Operations Safety Audit (LOSA) observation does. The aim of 
this chapter is to provide a working understanding of CRM and LOSA: how both 
programs integrate and support safety goals in aviation. 


10.1. CRM Description 


CRM covers a wide range of knowledge, skills and abilities including communications, 
teamwork, situational awareness, decision-making and leadership. CRM is the 
management of all resources—hardware, software and human-ware—to maximize 
safety and efficiency in flight operations. 


10.2. Company Operations Plan 


Commercial air carriers in the USA plan each of their flights to be in compliance with 
their Federal Aviation Administration (FAA) approved Operations Specifications. This 
ensures the flight is planned to the minimum compliant safety level. Air carriers then add 
tasks, decision aids, policies, procedures and other requirements, based on risk assessment 
and risk reductions, to improve the safety of every flight plan. The object of these additions 
is to account for all known system threats (weather, airport conditions, air traffic control 
delays, etc.) and prevent these threats from affecting the safety of each flight. The best 
operations plan is one where all threats are known and accounted for before the flight crew 
starts executing the plan. 


10.3. LOSA Definition 


LOSA is a safety data collection program that gathers frontline employee (flight crew) 
performance data during normal operations. It is designed to identify system safety 
issues, not to identify individual pilots or crews as being safe or not safe. It is used to 
diagnose the relative health of an air carrier’s level of safety in frontline normal 
Operations. 
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10.4. Line Operations Safety Audit (LOSA) 
10.4.1. The LOSA History 


The development of LOSA started with a desire by industry and the aviation research 
community to find better sources of safety data in normal flight operations. The standard 
safety data used prior to LOSA came from accident investigation and individual carrier 
required incident/event flight crew reporting. Commercial aviation accidents are rare, but 
often catastrophic with no survivors, making post-accident investigation difficult, at best. 
Required event reporting systems can put pilots in jeopardy and are historically under- 
reported. It would be far better to find ways to evaluate the flight crew’s performance in 
normal operations, before an accident happens, to gain insight concerning commercial 
aviation accident precursors. 

The University of Texas Human Factors Research Project (UTHF), headed by 
Professor Robert Helmreich, began normal flight operations monitoring in the early 
1980s. The project’s observations worked to evaluate Crew Resource Management 
(CRM), behaviors, skills and attitudes of flight crews as they flew their normal flights 
(Klinect, 2003). 

To ensure that flight crews are relaxed and that the observers would be unobtrusive, 
the pilots were told by a signed letter of agreement that all data would be de-identified, 
sent directly to the researchers at UTHE, and that there would be no jeopardy to the 
flight crews from the data. 

In 1995, Continental Airlines expanded their safety department, and being aware of 
Professor Helmreich’s work using normal operations monitoring, began contemplating 
a normal operations safety audit program. Continental wanted a program that would 
include collecting safety data on technical issues as well as CRM. A meeting was set 
between Continental Safety and Professor Helmreich in February 1996. The day before 
that meeting, Continental experienced a landing-gear up, landing accident at Houston 
Intercontinental Airport. (NTSB, 1996) 

While there were no fatalities and everyone walked away from the aircraft at ground 
level, the question was asked: How could a qualified flight crew land a perfectly 
good aircraft, with the landing-gear up, on a clear day, with very low air traffic volume? 
Before waiting for the answer, in the form of the NTSB Accident Report, Continental 
and UTHF came to agreement on a normal operations audit program. The program 
was favorably viewed by the FAA which approved FAA research grant funding to UTHF 
for the project. 

The program would use the UTHF methodology for normal operations monitoring 
which had previously been effective on CRM focused programs at Delta and other 


288 Chapter 10 - Crew Resource Management (CRM) and Line Operations Safety Audit (LOSA) 


carriers. However, it would be expanded to include data collection on flight crew 
performance concerning crew execution of the plan for their flight, and any errors the 
crew made in standard operating procedures, especially, if the flight environment 
became more difficult. 

The 1996 audit at Continental Airlines used 30 Continental instructor line captains 
and five UTHF researchers as observers. The audit collected data on all fleet type 
aircraft, over the entire route structure of the airline, in a three-month period. Eight 
hundred thirty-six flight segments were observed. The data took seven months to enter, 
collate and structure. Data analysis took an additional three months. The results were 
extremely illuminating. 

The data and analysis showed the good, the bad and the ugly. The good flights were 
flights where crew performance was excellent even when the operating environment 
got rough. Pilots on good crews did make errors but the crew discovered their errors and 
mitigated the consequences of those errors. They also stepped up to the challenges when 
the environment became nasty and good crews were proactive in their handling of 
system threats. 

The bad flights were flights with more prevalent errors that generally increased as the 
environment deteriorated. On these flights, the flight crew did not detect all their errors. 
Some of those errors may have led to a negative event except that some external system 
threats just went away (the weather cleared, or air traffic control provided a less difficult 
route, there was extra fuel on board, etc.). 

The ugly flights were flights where crews did not comply with procedures, 
made their own policies and disregarded rules and regulations, some vowing that they 
had a better way of accomplishing the plan. None of these flights operated below the 
minimum safety level, but the flight crews did not take advantage of the added safety 
provided by standard operating procedures (SOP) and good CRM. Intentional non- 
compliance would become a large safety target highlighted from the data. 

The 1996 audit at Continental stimulated other carriers, worldwide, to conduct 
similar audits partnered with UTHF and advanced the science of the normal 
operations monitoring process. Continental was the first repeat audit partner with 
UTHE accomplishing partial-system focused audits in 1997, 1998 and 1999. All of 
which led to the data and analysis structure of Threat and Error Management (TEM), 
which will be discussed later. 

With the TEM data taxonomy in place, the UTHF normal operations monitoring 
audit took the name of “Line Operations Safety Audit” (LOSA). The first LOSA audit 
took place at Continental Airlines in 2000. After that LOSA project, the FAA would 
only fund the research regarding the data and analysis, with no more funding for 
conducting LOSA projects at individual air carriers. 
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The funding of future LOSA projects would have to come from the carriers 
themselves. The growth of LOSA projects continued and required a separate entity 
to manage both the projects themselves and the continually growing archive database. 
The entity that evolved was The LOSA Collaborative (TLC) whose CEO was, and 
is, Dr James Klinect. Dr Klinect was the lead doctoral candidate under Dr Helmreich 
in 1996 when the Continental Airlines project first began and was the first to propose 
the TEM structure from the data. 

The agreement between UT and TLC ensures the openness of the normal 
operations monitoring program and data collection forms developed under FAA funded 
research. This has allowed carriers the ability to derive their own normal flight 
operations monitoring programs, which some have done. Eight normal flight operations 
monitoring audits (fewer than 4,000 observations) used the FAA research grant funded 
forms for data collection. Data collection forms and the data that were funded through 
the individual air carriers, remain the property of those individual carriers through The 
LOSA Collaborative. 

In 2005, LOSA was recognized as an FAA Voluntary Safety Program, with a FAA 
LOSA Advisory Circular, AC 120-90, following on April 27", 2006. As of November 
2006, TEM and LOSA concepts were added to several of the Annexes to the Convention 
on International Civil Aviation (Chicago Convention). In Annex 1 (Personnel Licensing), 
TEM is now a requirement for all pilot and ATCO licenses (standard). Annex 6 was 
amended to require TEM for all initial and recurrent flight crew training. In Annex 
14 (Aerodromes), the new Safety Management System standards highlight LOSA as a 
recommended practice for normal operations monitoring. (www.icao.int). 

To date, the LOSA archive database contains over 10,000 observed flight segments 
from over 50 audits and 35 different worldwide air carriers. The LOSA data have been 
used to indentify many issues; from checklist misuse and unstable landing approaches, to 
poorly designed operating procedures and outdated checklists/briefing guides/callouts. 
LOSA has also highlighted favorable and ineffective use of CRM behaviors/skills and 
attributes in all phases of flight. 

Significant changes to operations philosophies, policies and procedures have been 
driven by LOSA data in its TEM form. The data have spawned a new focus in training 
which is based on eliminating the consequences of threats and errors by using TEM 
countermeasures, CRM attributes and tools that bind human factors to specific task 
completion. 

The history of LOSA also shows the new emphasis that has been placed on 
safety change processes and safety management systems that involve everyone in the 
organization, at all levels, to work towards reducing the difficulty in the flight crew’s 
operating environment. (Helmreich, 2002) 
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10.4.2. The LOSA Process 


The LOSA process begins with an air carrier’s desire to obtain data on the crew 
performance as it applies to the execution of the flight plan in normal everyday 
operations. That desire has to manifest itself at a vice president level, or higher, for 
the project to take hold. Without support for the LOSA at that level, there is often 
too much resistance for project responsibility and funding or competition for control. 
Competition can exist between the functional areas of flight operations, flight standards 
and training and flight safety, fearing that any negative safety data will reflect poorly on 
any one of those areas of the organization. 

The difference between fear and anxiety is preparation. The fear of negative data 
turns to anxiety for the valid data, as the carrier prepares to conduct its first LOSA and 
begins to understand that there will be as much positive data as negative data resulting 
from the LOSA observations. 

There is a requirement to have both a LOSA oversight committee and a LOSA 
manager in place early in the project due to the pre-planning required. The oversight 
committee will determine the size of the LOSA in terms of how many dataset 
comparisons are desired, by aircraft fleet type, by crew base, geographical operations, or 
other comparisons. The committee will also select the observers and how they will be 
scheduled and compensated. The selection process chooses observers that have both the 
company’s recommendation and the pilot’s union recommendation from lists supplied by 
both the company and union. Committee coordination with the union will be similar to 
the departmental coordination between all other members of the oversight committee. 
The chairperson of the committee normally is from the sponsoring department. The 
committee’s decisions and tasking will become the focus of the LOSA manager for pre- 
observation requirement completion. The LOSA manager becomes the point of contact 
throughout the LOSA project including coordination with The LOSA Collaborative. 

The last requirement prior to the start of observations is observer training and 
calibration; both accomplished by The LOSA Collaborative. The most important 
training requirement is to ensure observers can write a complete narrative. The narrative 
describes how the crew performs in terms of executing the operations plan (flight plan) 
and how they handle external system threats and internally generated crew errors. 

How observers describe crew performance in terms of identifying system threats and 
internal crew errors, the responses that crews use after discovering threats and errors, and 
threat and error outcomes are the basis for observer calibration. These criteria will also 
be used to recalibrate the observers after their first few observations. Observers are 
scheduled to begin line observations immediately after training and calibration are 
completed. 
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The time period for conducting line observations varies depending on the number 
of segments required for data comparisons and the number of observers used. The 
observation period normally runs from one to four months. The data validation process 
begins as soon as the first data are received. The validation is completed after the data are 
reviewed and certified as accurate by the carrier. Data certification by the carrier is 
normally accomplished by members of flight operations and flight standards. 

Data analysis begins after certification. The analysis is accomplished using the 
structure of Threat and Error Management (TEM), developed by the University of 
Texas Human Factors Research Project in 2001. The LOSA Collaborative maintains the 
LOSA archive database which is used by carriers as a comparison to their data analysis 
results. The archive dataset currently contains more than 10,000 observed flight 
segments. 

In addition to the TEM structured observation data, LOSA also includes data from 
crew interviews regarding safety issues and/or survey data on pilot attitudes concerning 
organizational and safety culture, and resource management. All these data and analyses 
can be used to diagnostically uncover crew performance safety issues that currently exist. 
LOSA provides these as a snapshot of operations safety. 


10.4.3. Synthesis of the TEM Framework from the 
LOSA Data Tsunami 


The 836 flight observations from the 1996 Continental line audit used a data collection 
tool from UTHF named the Line/LOS Checklist, which effectively looked at CRM 
behaviors by phase of flight (Helmreich, 1995). The data collected described the 
effectiveness of crew performance in completing the required tasks of that flight’s 
operations plan. Added to the Line/LOS checklist was a Continental data collection 
form developed by Flight Standards and Training that collected crew error data relating 
to technical tasks such as handling performance (takeoff rotation angle and rate, climb 
profile, lateral, vertical and speed adherence, cruise altitude selection, the accuracy of the 
descent, arrival and approach handling to achieve a stabilized approach, and landing 
accuracy in terms of the landing touchdown zone). Also included were data on checklist 
accomplishment, briefing thoroughness, required altitude callouts and any limitations 
that were exceeded. 

The result of collecting all this information was a data tsunami. Both the Line/LOS 
data and the company technical data were analyzed. The analysis of the CRM behaviors 
data showed where flight crews had performed well, identifying issues that could or did 
impact the flight and how crews worked to reduce the difficulty back to normal through 
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effective teamwork. The analysis also showed where an increase in the difficulty of the 
system environment resulted in more errors, concerning technical task completions 
required during flight. All flight crew errors were captured in the analysis. 

Upon review, the data showed that crews with good CRM behaviors outperformed 
crews with poor CRM behaviors, especially when the system environment became 
more difficult. When the going got tough the good crews got going but the poor CRM 
crews got worse. The focus on correction for this finding was to develop tools that could 
help the crew determine when the system environment was moving to an “off normal” 
environment and how to bring it back to a normal state. 

Not until the Error Management Training for flight crews started, did the concept 
“there is no such thing as a normal environment”, come to light. Most everyone these days 
has flown a commercial flight to get somewhere. Our experiences range from flights that 
were on time, smooth, in good weather with our bags and dog waiting at baggage claim, to 
flights where the aircraft was late getting in, required maintenance to fix something, had to 
be de-iced because of the heavy snow falling, was so turbulent that the flight attendants 
never got up, had to hold over the destination but never got in due to heavy air traffic, and 
was diverted to an alternate airport, where the landing was firm and the airplane sat for 
three hours until an ATC clearance could be obtained to get back to our destination. 

So, which one is the “normal” flight? The answer is that there is no normal. There are 
flights with more or fewer threats that pop up during the flight, and there are flights with 
more or fewer crew errors made during the flight, but there is no normal! Therefore, 
flight crews can’t just accept the flight, thinking that what they get is the luck of the 
draw. They have to use their senses for the entire flight looking for threats and looking to 
trap their own errors when they occur, in order to manage those threats and errors to 
a safe conclusion without allowing the combinations of threats and errors to form an 
undesired state. 

The datasets from the Continental audits of 1997, 1998 and 1999, along with several 
foreign audits, clearly showed that the crew performance could be defined through the 
numbers of threats and errors experienced, and how those threats and errors were 
managed. The structure of the accident precursors and how they relate to each other and 
to accidents can be seen by reviewing Figure 10.1, UTHF Framework for Threat and 
Error Management. Of particular concern were the threats and errors that resulted in the 
formation of an undesired State. An Undesired State is defined as a state where the 
aircraft is in the wrong position or at the wrong speed or in a wrong configuration. 
Examples include wrong heading set for takeoff, wrong altitude set during descent, the 
landing gear not extended for landing, no flaps set for takeoff, or incorrect speed set for 
final approach. The list is extensive. Undesired states are to be avoided since they are the 
last state before an accident. 
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Every accident has a preceding undesired state; however, every undesired state does 
not result in an accident. The structure of the diagram in Figure 10.1 shows how threats 
can induce errors and how human errors can occur without a threat, but both threats 
and errors have to be managed before they become an undesired state or the undesired 
state is likely to become an accident. Undesired states are difficult to detect and can have 
a very short lifecycle before they manifest into an accident. 

It was the first few audits that provided the datasets that allowed the TEM framework 
to precipitate from the empirical data. The use of LOSA has brought about the focus for 
applying CRM. While developing training for TEM, countermeasures were explored as 
a proactive intervention to deal with threats and errors. Many CRM behaviors where 
considered but the basic TEM countermeasures that resulted came from defining the 
errors flight crews made when dealing with automation. 

Automation provides crews with the ability to be more precise and to work long 
calculations with ease. However, as powerful as the Flight Management Computer 
(FMC) is, it lacks one very important feature: it does not know flight crew intent! 
Because of that fact, the FMC is considered the dumbest crew member of the flight 
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Figure 10.2 Average Threat, Error and Undesired State data from the 10,000+ LOSA 
observation flight segments of the archive database 
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crew. The countermeasures developed for interfacing and using the FMC are simple. 
First, when asking the FMC to accomplish a task, verbalize to the other pilot the exact 
task you want the automation to perform. Then have the other pilot verify the question 
is correct. After verification, the execute function can be activated and then the last 
countermeasure, monitoring, can be used to ensure the desired intent is achieved. 

The most basic TEM countermeasures, VVM (Verbalize, Verify and Monitor), 
developed for use with the dumbest flight crew member happen to work well with the 
smarter crew members, the humans, too. By verbalizing, we start to build the CRM 
attribute of Communications. It is critical to verbalize the exact information of concern 
to the appropriate flight crew members or outside agencies. That is what builds the 
second CRM attribute of Coordination/ Teamwork. The information passed by 
verbalization updates the crew member(s) to ensure their individual mental models are 
equal to the shared mental model and that the shared model is equal to reality. This 
defines Situational Awareness (SA) as the third CRM attribute, and is required if flight 
crews are to make good and safe decisions. For safety in line operations, good decision- 
making is the fourth CRM attribute and safety goal. See Figure 10.3 (Threat & Error 
Management Hierarchy of CRM attributes ...). Higher level CRM behaviors of 
leadership, modeling, mentoring and others are designed to bring improvement to 
VVM skills and to foster crew learning. 

The last group of countermeasures includes the tools used to marry human factors 
knowledge and performance with specific technical task accomplishment. These tools 
are briefings, callouts, bottom lines, checklists, criteria matrices and standard operating 
procedures. If you want to communicate the plan you use a briefing, if you want to 
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Figure 10.3 Threat and Error Management hierarchy of CRM attributes and TEM 
countermeasures that build to safe decision-making 
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verify that your altitude for climb or descent is near you use a callout, if you want to 
ensure all required tasks are completed before takeoff you design a checklist. The tools 
are as important as the CRM skills or the technical skills of task accomplishment. Tools 
are used to reduce the difficulty of the frontline employees’ work environment, which 
increases the ability of the crew to avoid and manage the accident precursors. The entire 
organization wants to reduce environmental difficulties since the frontline is doing the 
work at the sharp end of the spear. 


10.5. Flight Crew Performance 
and Procedural Drift 


CRM and TEM data from the LOSA observations, along with data from the crew 
interviews and surveys, showed that there is a crew performance negative drift component 
that affects compliance with the company’s guidance, policies and procedures. Drift is 
related to the time between currency events and also to the time between training and 
checking events. The more recent the experience and training for that event, the closer 
the performance was to standard. 

Flight crew performance relative to standard operating procedures appeared to 
decline with time (drift) for three reasons. First, is unintentional drift, where deviations 
are generally minor. When the flight crews unintentionally drif, and they identify 
the drift, they immediately self-correct. The second form of drift concerns following the 
“norm” instead of the standard procedure. It is equivalent to driving a motor vehicle 
over the posted speed limit to keep up with traffic because everyone else is speeding. The 
correction for drift due to a norm is training to show why the speed limit is there and 
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practice in obeying that limit. Training corrects this form of drift because those involved 
want to comply with the standards. The third form of drift is intentional non- 
compliance. It is the worst and most insidious. 

Intentional non-compliance is reckless behavior. It stems from a belief that guidance, 
policy and procedures were meant for the weakest individual and not a skilled 
professional like me. “I have a better, faster, smoother way of doing things.” “I liked the 
way we did it at XYZ, so I’m doing it that way.” “I have more experience than the 
people who wrote these procedures and I know best.” 

Intentional non-compliance cannot be tolerated by anyone, especially other crew 
members. Discipline is required and appropriate, within a just-culture, to deal with 
intentional non-compliance. The only requirement before applying discipline is to 
answer the question: Is the guidance, policy or procedure reasonable? When intentional 
non-compliance occurs on a frequent basis it tends to show an unrealistic, hard to 
comply with procedure or that the entire operation is too procedural and does not allow 
enough crew flexibility that is reasonable and safe. All three forms of drift were observed 
in the LOSA observations and became even clearer in the crew interviews. 

The disturbing findings from the LOSA archive database show that crews observed 
in one or more intentional non-compliance errors had three times the number of 
mismanaged threats and errors and also a higher number of undesired states than crews 
without intentional non-compliance errors. This is another reason why intentional 
non-compliance cannot be tolerated. 


10.6. The Safety Change Process and Safety 
Management Systems 


As mentioned earlier, LOSA is a recommended voluntary safety project approved by the 
FAA and ICAO. It provides a detailed look at flight crew performance in normal 
operations. But data and analysis aside, there will be no reduction in the number or 
severity of the LOSA detected accident precursors without action to develop safety 
changes. Every organization needs a safety change process that functions in a continuous 
cycle beginning with measurement of safety, where valid data are collected and starts 
again with measurement of the changes implemented from the last measurement. The 
entire cycle consists of safety measurement, data analysis, safety target identification, 
proposed changes to reduce the risk of the safety target, prioritization of risk reductions, 
funding of the changes and implementation of the approved and funded changes. Since 
auditing of the results is a function of the next LOSA, the audit process is already in 
place. Auditing of Threat and Error Management is also in place since any level can ask 
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levels below what threats they are working to reduce and a response of “I don’t know” 
means a process breakdown at the lower level. Change is inevitable and safety change in 
normal operations is no exception. 

Safety Management Systems (SMS) provide a means of conducting operations risk 
assessment, operations task risk reduction and ongoing risk management through 
programs like LOSA. SMS involves every level of the company by placing requirements 
for safety processes that keep all levels of the organization aware of and involved in safety 
management. SMS keeps the middle management zone from spinning the direction 
from the top and blocking information from the frontlines on the way to the top, and 
truly holds everyone in the organization accountable for safety change and awareness. It 
makes the regulator’s task of ensuring regulatory compliance easier because the 
company’s focus is on safety change; being better tomorrow than you were today and 
having the data to prove it. It is no longer a business of avoiding regulatory fines for 
violations to the regulations; it is process management for safety improvement. 

LOSA and CRM are integral parts of safety improvement within SMS and continue 
to be linked together. 


10.7. Summary 


= CRM is defined as the use of all hardware, software and human-ware to manage all 
resources and achieve a safe flight. 


m LOSA is a safety data collection program based on observations from the cockpit 
jump-seat by trained pilot observers on normal flight operations. 


m The LOSA observations are at no jeopardy to the flight crew as the project looks at 
how well flight crews perform in managing the system threats and flight crew 
errors that are present on almost every flight. 


m The history of LOSA starts with the Line/LOS-CRM behavioral marker data 
collection by observation, first accomplished by the University of Texas Human 
Factors Research Project. 


m The Threat and Error Management framework came together from the precipi- 
tation of empirical LOSA data. 


m The LOSA Archive database is continuously updated with the finish of each 
LOSA project (10,000+ flight segments from 50+ separate audits for 35+ 
worldwide air carriers). The database is maintained by The LOSA Collaborative. 
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mw TEM data provide a focus for application of CRM. 


m= TEM and CRM countermeasures provide a hierarchy of attributes that when 
achieved simplify and are a catalyst to successfully managing the accident 
precursors. 


m Tools that marry CRM skills to operations plan task accomplishment are critical in 
managing accident precursors. 


m Intentional non-compliance is the most serious form of performance drift and data 
show three times the number of mismanaged threats and errors for crews that 
participate in intentional non-compliance. CRM and LOSA are forever strongly 
linked in the TEM taxonomy. 


Author's Perspective 


This chapter references only materials from the members of the University of Texas 
Human Factors Research Project, the FAA, the NTSB, and ICAO. The reason for this is 
because the actual LOSA data, that includes CRM data and technical data, are 
proprietary to the individual air carries that participated in LOSA audits. My 
involvement in developing LOSA, Threat & Error Management and CRM was to assist 
those conducting research by providing a test bed in actual operations, then using the 
results to stimulate safety changes by the organizations conducting the LOSA audits. 

All six of the LOSA audits that I managed and the seventh that I consulted on at 
Continental Airlines were different. The time span for those seven LOSA audits ran 
from 1996 through 2008 in which time there were numerous changes in personnel at all 
levels of the organization. I want to thank all the Flight Operation’s, Flight Standards & 
Training and Safety & Regulatory Compliance Directors and Vice Presidents for their 
support and belief in the LOSA and Safety Change Process, especially during the ups 
and downs of the business cycle. While the structure of funding the LOSA changed with 
every audit, there was never a question of if the LOSA was going to happen; just 
a question of how it would be accomplished. From my communications with LOSA 
managers at other air carriers, I find the same flexibility is required by every air carrier. 
The data and analysis from LOSA can be obtained in no other way than by normal 
operations monitoring. 

I encourage all who read this chapter to contact the LOSA managers of all LOSA 
accomplished air carriers to validate and update the perspective I have given in this text. 
Most carriers are willing to openly share their findings because they feel a commitment to 
aviation safety in general. To obtain the latest information concerning LOSA, TEM and 
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their relationship with CRM, contact Dr. James Klinect, CEO of The LOSA 
Collaborative, through his website. He has been involved with LOSA and TEM from the 
beginning and has a perspective supported by past and current data that no one else has. 


10.8. Questions and Answers 


1. 


10. 


What is CRM? 
It is the management of all hardware, software and people-ware to achieve a safe 
and efficient flight. 


. What is LOSA? 


It is a safety data collection program for normal operations monitoring, looking 
for system problems that the crews must manage. 


. Is LOSA recognized by the FAA? 


LOSA is recognized by the FAA as a Voluntary Safety Project and is also 
recommended as an ICAO best practice. 


. What data collection types are used in LOSA? 


LOSA uses non-jeopardy observations, crew interviews and crew surveys as data 
collection types. 


. What constitutes a “normal” flight? 


There is no such thing as a normal flight. 


. What defines the dumbest crew member on the flight? 


The dumb automation, because it cannot know the crew’s intent. 


. What is the highest level CRM attribute that TEM looks for in safe operations? 


Building on Communications, Coordination/teamwork and Situational Aware- 
ness is the focus CRM attribute of decision-making. 


. What is used to ensure good CRM behavior skills are directly tied to operations 


plan technical task accomplishment? 
Tools such as briefings, callouts, checklists, limitations, bottom lines and SOP. 


. What is the worst form of performance drift? 


Intentional non-compliance; it is reckless behavior. 


What is the historical data supported risk if you choose to intentionally non-comply? 
A three times higher rate of threat and errors mismanagement. 
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Introduction 


During the STS-87 space shuttle mission in late 1997, a satellite was deployed without 
first being activated. A detailed post-flight analysis and investigation of the satellite 
deploy found that several operational errors had occurred. 

Up until this time the responsibility for addressing Crew Resource Management 
(CRM) issues among NASA space shuttle crews was largely deferred to the designated 
commander of each mission. Assigned crews train extensively for one to two years and 
the prevailing thought was that “crews naturally gel” into well-coordinated teams over 
time. One of the major problems that reinforced this thought was that the instructors 
on each crew’s training team were not versed in the best methods used to actively train 
CRM issues or its benefits. In early 1997, several instructors from NASA’s Space Flight 
Training Division began to challenge this paradigm. These folks had years of previous 
experience teaching CRM principles within both the civilian and military sectors. 
They began to ask management: “How can we assure that crews actually ‘gel’ if we do 
not monitor the efficacy of CRM skills or the resolution of CRM issues that arise 
during training?” The short answer was “we could not.” As a result, a proposal was 
introduced to create a formal human factors curriculum for astronaut crews based on 
the CRM training currently being implemented in the aviation industry. Fortunately, 
the Chief of the Astronaut Office was a staunch proponent of this proposal and helped 
advocate to management that some form of CRM training for astronauts was long 
overdue. 


11.1. The Crew 


If we define a “crew” as the end-users of CRM training and the principal 
decision-makers, then there is a fundamental difference between the aerospace and 
aviation industries. Unlike our commercial aviation counterparts, shuttle crews 
rarely make autonomous decisions. Compare the takeoff decision of a commercial 
airline to that of the launch decision of the space shuttle. In the case of the airline 
flight crew, the captain is the sole decision-maker. The crew completes all pre- 
flight planning, crew briefings and checklists, obtains the proper ATC clearances, 
confirms the vehicle’s mechanical status and advances the throttles to initiate the 
takeoff. If the caution and warning system annunciates a problem, during the 
takeoff roll, the co-pilot confirms the problem and the captain is faced with the 
immediate decision to abort or continue the takeoff. The space shuttle commander 
does have the authority to take actions to assure the safety of the flight crew, but in 
practice this would only be exercised in an emergency situation where 
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communication with the flight director had been lost. During shuttle launch 
operations, the “crew” is comprised of hundreds of people. Largely due to the 
shuttle’s complexity and the fact that the flight crew is “blind” to many critical 
parameters, the shuttle crew, launch team at Kennedy Space Center (KSC), 
Mission Control Center (MCC) flight control team at Johnson Space Center (JSC) 
and Mission Management Team (MMT) all play critical roles in the decision to 
launch safely. Launch procedures and flight rules provide the necessary guidance to 
the launch team and MCC for many problems during this critical time. However, 
management’s role should not be overlooked. When a problem arises that is not 
well defined, it becomes the responsibility of the MMT (senior management 
personnel) to understand the issue, scrutinize the developed flight rationale to 
proceed and provide a final go/no-go decision. The expansion of the “crew” to 
include both the flight crew and all ground-based teams holds true for every phase 
of shuttle operations. 


11.2. Defining the Problem 


Research consistently reports the most dominant causes of accidents are due to 
either technical or human factors failures. Of these two, human factors failures 
account for about 70-80% of the cases (Cooper et al., 1980). To refine the problem 
further, research within high reliability organizations shows the human component 
of accident causes can generally be divided into either safety culture or operator 
error. Surprisingly, organizations identified as having a weak safety culture are 
responsible for about 80% of the human factor-related accidents (Flin, 2005). As 
a scientific and engineering-based community, we began to ask ourselves 

“Where do we spend the majority of our energy; fixing hardware or improving 
ourselves and our team effectiveness?” The answer was clear; historically, 

NASA poured nearly all of its resources in this area identifying, designing, 
correcting and controlling technical problems. This should not be a surprise. 
Solving tough technical problems is what NASA does best. While the critical 
importance of applying our engineering prowess to solve technical issues should not 
be minimized, as risk managers we should also be asking, “Is it prudent to spend 
such a disproportionate amount of effort to protect 20-30% of risk while 
inadequately addressing the other 70-80%?” Based upon the composition of the 
spaceflight “crew” and realizing where we were most vulnerable, it was obvious 
that the real problem was not a shuttle flight crew issue but rather a NASA team 
problem. 
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11.3. A Vision and Constraints Imposed 


It became readily apparent that we would fall short of helping solve the real problem 
if the training were limited to astronauts alone. To be truly effective and address the 
more systemic problem, the CRM training needed to include the entire team: 
flight crews, mission controllers and senior NASA management personnel alike. 
Unfortunately, the reality within the Space Flight Training Division was the existing 
government contract provided for astronaut and flight controller training, but did not 
provide for management training. While the operational errors mentioned in the 
STS-87 investigation included communication errors between the flight crew and 
ground controllers, management was convinced it was a flight crew training issue 
alone. In light of these constraints, the team adopted the strategy first to address the 
immediate need and direction of providing CRM training to NASA’s astronauts. As 
the training program matured, we would invite members of the flight control 
community to attend the classes. In order to further expand the training, we 
recognized the importance of gaining an advocate who appreciated the value of the 
training and who had the authority to help spearhead the effort in their area. For 
quite some time, the thought of providing CRM training to senior NASA 
management was off the table. It would take eight years, the Columbia Accident and 
the move of two members of the original development team to other parts of the 
Space Shuttle Program before formal CRM training would be given to NASA and 
contractor management personnel. 

With the original mandate in hand, the development team went to work on 
satisfying the astronaut training requirements. Knowing we would be setting our sights 
on other communities in the future, the team changed the program name from Crew 
Resource Management (CRM) to Space Flight Resource Management (SFRM) to 
subtly emphasize what was being addressed was not only a crew training issue, but one 
that included every member of the larger NASA team. 


11.4. Operational Insights 


Members on the SFRM development team were fully aware of the research-based 
recommendations being advocated by the academic community. What was not 
known was how successfully the operational communities, external to NASA, were 
implementing these recommendations. In order to gain this insight, the development 
team visited two major commercial air carriers and two nuclear power companies. 
The primary objectives were: (1) discover first hand, the nature and content of 
each organizations CRM program, (2) capture through open dialog, what was 
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working within each program, and identify the specific reasons for success and (3) 
learn what mistakes were being made during program development and the impact of 
each unsuccessful outcome. 


11.5. SFRM Training Philosophy 
and Development 


Based on the academic research and operational data gathered, the development team 
agreed that to be accepted within the current NASA spaceflight training environment, 
all SFRM training would be based on an error management philosophy; in addition, it 
would be skill based, operationally relevant, fully integrated within technical training, 
and continually reinforced. Reinforcement proved to be the largest challenge. While the 
SFRM development team would deliver the classroom instruction, the reinforcement of 
SFRM training would take place during every shuttle astronaut simulator training 
session. The instructor experience base with respect to CRM concepts and training 
methods was extremely limited. Before the SFRM team could deliver effective training 
to the astronaut corps, baseline instruction of SFRM concepts for more than 250 
instructors was required. 


11.6. Courseware Design and Instructor 
Training 


Three classes were developed to address the instructors’ training needs. The SFRM 
Overview course was designed with two primary objectives in mind: (1) increase each 
participant’s awareness of SFRM concepts and (2) standardize the vocabulary and 
application of SFRM training throughout the training division. Six categories, or 
performance elements, were chosen that would establish the framework for the set of 
skills most applicable to NASA: Command, Leadership, Communication, Workload 
Management, Situational Awareness and Decision-Making. All of the SFRM classroom 
lessons were instructionally designed to limit lecture and maximize student 
participation. To further enhance discussion, class sizes were limited to 30 students, with 
a targeted optimum class size of 18-20. The SFRM team instructor presented the 
general concepts associated with a particular performance element and then facilitated 
a discussion of how these concepts apply operationally at NASA. Next, a set of 5-6 
skills/behaviors were introduced that tied back to each element being discussed. Shuttle 
spaceflight training instructors would be ultimately responsible for SFRM skill 
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observation to the same level of expertise they already possessed within their technical 
disciplines. To address this training requirement, the development team used relevant 
video clips from popular movies. Each video illustrated the active use of the SFRM 
element skills being discussed. After viewing each clip, students were asked to identify 
which of the skills were present and how they were used. The most common feedback 
from class critiques indicated that this approach made the SFRM concepts and skills 
come alive and fostered the notion that with practice they could actively observe SFRM 
and technical skills with equal proficiency. 

Building upon the SFRM skill observation that was introduced during the 
Overview class, instructors attended SFRM Applied Methods. The primary objective 
of this class was to sharpen and refine the student’s SFRM skills observation by 
replacing movie clips with actual spaceflight missions and training session examples. 
This approach not only brought operational relevance to the training, but gave 
students a clearer understanding that as instructors, their observations, coupled with 
effective debrief skills, would become the principal vehicle to reinforce SFRM 
training. 

Lastly, the SFRM Facilitation class rounded out the instructor training requirements. 
This class provided instructors with the knowledge of how to fully integrate the SFRM 
training within the existing astronaut technical curriculum. Instructors evaluated actual 
training debriefs against a set of SFRM debrief criteria. Flight crews are expected to: 
(1) raise and initiate discussion of SFRM topics directly with each other, (2) critically 
analyze the impact that SFRM skills had during the training scenario, (3) develop and 
implement specific actions to improve those skills deemed less effective and (4) identify 
and reinforce skills that added to the crew’s effectiveness. While using more instructional 
methods to cover technical issues is appropriate, instructors discovered that by taking 
a facilitative approach when debriefing SFRM topics flight crews learned more from 
each training session, and developed critical self-critiquing habit patterns. To this end, 
instructors were introduced to specific questioning strategies designed to facilitate an 
effective crew-centered debrief. 


11.7. Astronaut SFRM Training 


Every astronaut candidate (ASCAN) receives the SFRM Overview and Applied 
Methods classes as part of their initial training curriculum. These classes occur during 
the first month of space training to stress the importance of the SFRM skills. How 
effective SFRM skill usage will enhance their individual and team performance is 
specifically stressed. There is an enormous difference between knowledge of SFRM 
concepts and skills and a commitment to the SFRM training philosophy. Each ASCAN is 


Crew Resource Management: Spaceflight Resource Management + Chapter 11 


challenged to make a personal commitment to SFRM skill reinforcement. To make such 
a commitment, each astronaut is expected to demonstrate discipline and resolve by 
integrating SFRM discussions during every technical training opportunity. 

As flight crews are assigned to a specific mission and begin their training, the first 
assigned crew activity is an SFRM Refresher class. The primary objective of this class is 
to give the mission commander a forum for establishing open communication and 
positive crew climate among crewmembers. The class is divided into two parts, each 
stressing a particular objective. The first part addresses the inter-relationship of the six 
SFRM performance elements. Crews then listen to an audio recording of MCC 
communications during an actual launch addressing an in-flight anomaly. The crews are 
then asked to identify and evaluate the effectiveness of the SFRM skills used by the 
MCC flight controllers. Specific emphasis is placed on assessing the communication 
skills used, and how those skills led to a heightened level of situational awareness among 
the flight crew/flight control team. This discussion serves as a springboard where each 
crewmember then shares an example drawn from their experience of either effective or 
ineffective SFRM skill usage. To ensure that these discussions remain operationally 
relevant, the crewmembers are asked to elaborate on how each lesson learned may be 
applied during the assigned crew training they are about to begin. One first-time 
crewmember stated an incident where he was involved in a near mid-air collision during 
a previous military training flight. The primary reason for the near miss was due to poor 
communication between himself and a much more seasoned pilot who was his 
instructor at the time. He got distracted during a critical moment and assumed his 
instructor (who was also distracted) had the intercepting aircraft in-sight. Within 
seconds they were staring at an aircraft rapidly getting larger in their windscreen. Evasive 
actions saved their lives that day but they came within 100 feet of certain death. After the 
pilot shared this experience, the shuttle commander reminded the entire crew that there 
will be many times during training and their mission when it will be very easy to become 
distracted from critical tasks. He warned everyone not to assume that he has the “big 
picture” all the time. ““When you see something...communicate. When in doubt, speak 
up and communicate that too! The mere fact that I’m the commander does not make me 
immune from becoming distracted.” The entire crew then began to develop strategies 
and communication techniques to remain vigilant during critical events. In the second 
part of the class, the instructor explicitly defines and establishes expectations with regard 
to debriefing SFRM topics. It is left up to the shuttle commander and crew to decide 
how they choose to incorporate the SFRM topics and issues that arise. However, the 
crew is reminded that it is not sufficient to discuss topics as a crew and critically analyze 
the effectiveness of their skill usage, only to state that they will “not make the same 
mistake(s) in the future.” In such cases, the training team will challenge the crew to 
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develop and state specific strategies to be employed. During the next training 
opportunity, the training team will monitor the efficacy of these strategies and provide 
feedback to the crew when necessary. 


11.8. MCC Flight Controller SFRM Training 


Because shuttle flight crews rely on the direction and expertise of mission control as their 
primary external resource, bringing SFRM training to the flight control community was 
the next logical step for expansion of the SFRM program. Not until the instructor 
training was complete and astronaut training well established did the SFRM team begin 
to solicit flight control managers to audit the SFRM classes. It did not take more than 
a few sessions before many managers began to ask the SFRM team to develop a class 
tailored for flight controllers. 

To baseline those already fully certified having a basic understanding of CRM 
concepts, the SFRM Applied Methods for MCC course was developed. The primary 
objectives of this class are to operationalize the essential concepts behind the SFRM skills 
and to practice SFRM skill observation, assessment and debrief critiques. This class is 
taught from a flight controller perspective, and combines the core content of the SFRM 
Overview (less the movie clips) with all of the actual space flight examples from the 
SFRM Applied Methods course. As a result of the support and acceptance from NASA 
management, the flight control community included SFRM training as a required 
course of instruction for all those seeking certification as a NASA shuttle flight 
controller. All new hires aspiring to become flight controllers attend the SFRM 
Overview class during the first month of working at NASA and take SFRM Applied 
Methods for MCC as they near certification. 


11.9. Space Shuttle Maintenance SFRM 
Training 


Through the SFRM team’s invitation to reach out to interested parties, word soon 
spread to other NASA organizations of the SFRM program and its positive results. 
Looking for ways to engrain CRM principles into the space shuttle vehicle maintenance 
processes, members from the Human Factors Engineering Group at the KSC attended 
the SFRM classes. Members from the SFRM and KSC development teams worked 
together to discover ways to bring the SFRM training to vehicle maintenance personnel. 
The KSC training group provides a customized version of the SFRM Overview to 
newly hired shuttle maintenance personnel. 
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11.10. SFRM for Space Station Flight Control 


While the shuttle SFRM training was well under way with the primary focus of SFRM 
reinforcement and debrief facilitation, International Space Station (ISS) instructors 
began taking the SFRM Applied Methods for MCC and Facilitation classes. The SFRM 
team was comprised of members from the Shuttle Guidance & Control/Propulsion 
group within the Space Flight Training Division. As space flight examples were 
developed for the Applied Methods and Facilitation classes, the team drew from the 
flight and training examples that were germane to shuttle operations. Additionally, to fit 
the case studies within a reasonable class time, examples from dynamic flight phases were 
targeted because they developed much more quickly than those during more quiescent 
orbit operations. Clear differences in operational philosophy exist between the Space 
Shuttle and ISS Programs. As the class composition began to be more heavily weighted 
toward ISS personnel, it became apparent that students were losing the desired 
operational relevance from the training. ISS instructors formed an ISS SFRM working 
group and began researching ways to tailor the SFRM training to ISS operations. 

Drawing from the original SFRM team’s lessons learned and training philosophy, ISS 
instructors began to mirror the initial development efforts used by the team eight years 
earlier. In late 2006, the team gathered ISS flight controller survey data and began 
industry benchmarking several 24/7 operational control centers. The data enabled the 
team to develop a customized ISS SFRM model. The genesis for SFRM training within 
the ISS flight control community came when the team was given direction from NASA 
management to examine ways of increasing the efficiency in ISS flight controller 
training (Baldwin, 2008). 


11.11. SFRM for Management 


In 2004, a year after the Columbia accident, two members from the original shuttle 
SFRM team changed jobs from shuttle spaceflight training to shuttle Safety & Mission 
Assurance (S&MA). As contractors charged with providing NASA with independent 
safety oversight, shuttle S&MA also provides assessments, briefings and support to 
program-level NASA management. 

After the release of the Columbia Accident Investigation Board (CAIB) report, the 
Space Shuttle Program (SSP) began an intensive training program for Mission 
Management Team members that included outsourced leadership, critical decision- 
making and CRM classes. Two and a half years later, the MMT that emerged during the 
STS-114 Return-to-Flight mission represented a team far superior to what they had 
been prior to the Columbia accident. 
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After STS-114, the SSP deputy manager (MMT chair) began to look for ways to 
build upon the team’s marked improvement. For the most part, MMT members were 
aware of what team skills and behaviors were required of them; what they lacked was 
how best to sustain the progress that had been made. Seeking to obtain higher 
operational relevance in the aviation-based CRM class, the SSP deputy manager turned 
to S&MA’s former shuttle SFRM team members to better align this class toward shuttle 
operations and the NASA/MMT environment. An MMT SFRM: Concepts and 
Implementation class was developed specifically tailored for senior management and the 
unique role of the MMT during shuttle missions. Relevant sections from the shuttle 
SFRM Overview, Applied Methods and Facilitation classes were used, and all case 
studies and exercises were replaced with video and audio examples taken from actual 
MMT training sessions. Even more critical was the great emphasis placed on debriefing 
and self-assessment skills designed to reinforce the team’s SFRM skills and lessons 
learned. 


11.12. SFRM: Safety Culture 


A recurring theme in the literature is that organizations with effective safety cultures 
share a constant commitment to safety as a top-level priority; and this commitment 
permeates the entire organization. Some of the common components are the following: 
acknowledgment of the high-risk, error-prone nature of an organization’s activities, 

a blame-free environment where individuals are able to report errors or close calls 
without punishment, the expectation of collaboration across ranks to seek solutions to 
vulnerabilities, and a willingness on the part of the organization to direct resources to 
address safety concerns (Taylor and Rycraft, 2005). 


11.13. NASA'S Safety Culture Challenge 


The space environment is an unforgiving place in which to operate. For NASA to carry 
out its manned-space mission it is accepting a 1 in 80 chance of losing a national asset 
with the lives of seven astronauts on the line every time the shuttle launches. Accepting 
risk is part of the space business. The most important aspect is to fully understand the 
risks being taken—get it wrong, and we are at an even greater risk of having a “bad day.” 
In all, though, there is little debate that NASA has an outstanding safety record. 
Yet, NASA has not been without its critics when it comes to the agency’s safety 

culture. During the years after Apollo 1 and Challenger, much was done in the way of 
process improvements and organizational restructuring but little in terms of applying 
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a long-term solution to the human-error problem. Many point to Challenger and 
Columbia as being organizational accidents. While there were clearly organizational 
pressures present, this does not provide a full explanation for the agency’s inability to 
sustain a healthy safety culture in the years between the two accidents. Organizational 
pressures such as production or operational schedules (real and perceived), a wavering in 
the agency’s vision, the organizational structure (especially that of safety) and fiscal 
constraints establish the conditions in which teams operate within the larger framework 
of the entire organization. All represent a threat to a team’s current safety climate. The 
reason for the decline of the organization’s safety culture hits much closer to home. It is 
what we actively do or fail to do as individuals and as teams to strengthen our safety 
culture that determines if we will allow ourselves to succumb to the threats we wrestle 
with every day. Although it may seem to be the case at times, we are not at the mercy of 
the external pressures that exist. Our daily actions and behaviors that work to embolden 
our team effectiveness and resilience act as a powerful counterforce to the organizational 
pressures that are part and parcel to our environment. 

Recent efforts in NASA to regain and sustain a healthy safety culture include actively 
fostering the following skills and behaviors: leadership, teamwork, communication, 
understanding risk, decision-making, trust, accepting team and individual responsibility 
and demonstrating resolve toward continual improvement. Although these provide 
teams with the tools necessary for minimizing the likelihood of making errors, it has 
been shown that a team’s willingness and commitment to use and sharpen them through 
constant evaluation have the largest impact on developing exceptional teams and 
sustaining a strong safety culture. 


11.14. Debriefs and Self-Assessments 


Crew or team-centered debriefs and self-assessments are the cornerstone in SFRM 
training. Introducing this method of SFRM skill reinforcement for astronauts and flight 
controllers was not difficult. For these groups it simply meant integrating SFRM topics 
into an already well-established debrief process. However, prior to MMT training, the 
notion that a management-level team would pause to evaluate their decision-making 
process did not exist. Regardless of the application, sound leadership traits such as 
maintaining a positive team atmosphere, adopting a learning mentality and 
demonstrating professional maturity are paramount during all debriefs. The team leader, 
in particular, must establish and maintain a positive atmosphere. It is important that team 
members understand that debriefs and self-assessments are never about pointing fingers 
or assigning blame. It is simply the vehicle used to ensure an open and honest dialog for 
the primary goal of improving individual and team performance. Adopting and fostering 
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a learning mentality shows a commitment to truly learn from shared experiences. The 
very fact that teams take the time to participate in the debrief and self-assessment process 
demonstrates this resolve. Lastly, demonstrating professional maturity is critical when 
conducting team-centered debriefs. Being able to admit your weaknesses and 
shortcomings to others requires a great deal of courage and integrity. Conducting 
self-assessments is an integral part of the debrief process; it reinforces all of the 
above-mentioned leadership traits and demonstrates a commitment to place the team’s 
improvement over that of personal egos or agendas. Self-assessments remind members 
their actions and behaviors have a direct impact to the success or failure of the team. It 
provides the opportunity to demonstrate personal accountability and commitment to 
the team’s success. Additionally, this practice has the profound effect of building trust and 
respect among team members. 

In the case of a team-centered debrief, it is the members of the team who take on the 
role of analyzing their performance. The goal is self-discovery, self-correction and 
reinforcement. Members in the aerospace industry and those within high reliability 
organizations tend to measure success one mission or milestone at a time. This measure 
of success can place such a team at an increased risk of developing an outcome bias 
during the debrief process. This bias occurs when the outcome of a decision is used to 
judge the quality of the decision process or team effectiveness (Reason, 1997). With 
such a bias, teams assume good outcomes are the result of effective decision processes 
and bad outcomes the result of flawed processes. However, research shows no such 
correlation exists. If teams allow themselves to hold this assumption, they run the risk of 
failing to address cases where in spite of their weaknesses, they got lucky and a positive 
outcome resulted. By focusing debriefs on how and why the team arrived at a decision 
the danger of this bias can be minimized. 


11.15. A Success Story 


The SSP MMT’s efforts to embrace the value of team-centered debriefs and self- 
assessments has brought about a positive cultural change within the larger NASA team. 
The turning point occurred when the MMT chair established his expectations during 
MMT debriefings to the MMT membership. This wasn’t done by memo but through 
mentorship and superior leadership—by modeling the behaviors he expected his team 
members to emulate. For perhaps the first time, members of the MMT witnessed 

a senior manager admitting his own mistakes and the effect it had on the decision- 
making process and team effectiveness. Nearly every member of the MMT followed the 
chair’s example. As a result of this action and many more that followed, MMT debriefs 
have been consistently characterized as brutally honest and open, with egos put aside 
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both from a team perspective and in each team member’s assessment of his or her own 
performance. The impact of these measures has been profound. The shuttle MMT 
membership has shown a steadfast commitment to implementing continual 
improvement. They have adopted a learning organization mentality where every 
decision and team interaction, whether occurring during a simulation or actual mission, 
represents an opportunity to learn and improve both as a team and as individuals. They 
are especially sensitive to identifying areas where repeated successes could lead to 
complacency. Dissenting opinions are encouraged, solicited and routinely voiced. 

Though lessons learned databases are useful in capturing the “historical record” of 
errors, they have a poor track record in being able to raise the level of awareness 
sufficient to prevent problems from recurring. The shuttle MMT has taken a fresh 
approach to recalling previous lessons learned. The MMT realized that their team 
debriefs and self-assessments offered a far more effective tool because lessons learned are 
continually reviewed and give team members an opportunity to be directly accountable 
and to actively implement specific solutions for improvement. In an effort to keep from 
forgetting lessons between missions, the MMT conducts a mission pre-brief prior to 
each shuttle flight. These briefings help remind the members of previous lessons learned 
and improvement strategies to keep from “running over the same land mines.” 


11.16. SFRM Lessons Learned 


Astronauts, flight directors and program managers are not exactly pushovers, and 
achieving sustainable cultural change for such a population is not a trivial task. The most 
important step is gaining the support of the organization’s authoritative figure (Pruyn & 
Sterling, 2005). For those who already understood the value of CRM/SFRM training, 
gaining support was rather straightforward. For others it took time for them to see what 
effect the training had on other groups before they were convinced that SFRM was 
going to be a priority for their organization. If the organization’s “commander” supports 
the effort, over time, so will the organization. The “bottom-up approach” of trying to 
lead cultural change from below—while possible—is about as much fun as trying to 
rescue a beached whale by single-handedly pushing it back into open water (Pruyn & 
Sterling, 2005). 

Another SFRM lesson learned is that training programs tailored to match the culture 
of an organization and its operations possess the greatest likelihood for success. Keeping 
in mind the research done by academia and the operational implementation from many 
industries allowed the SFRM team to avoid replicating identified pitfalls and to 
capitalize on successful approaches. In some cases they were a good fit and in others 
unique solutions were necessary. Making the classroom instruction operationally 
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relevant to the student population enables the participant to see more clearly how 
SFRM training applies in their daily work environment whether it be a shuttle flight 
deck, control center or conference room. The most effective way to bridge philosophy 
to actual operations is to find examples that come from one’s own organization. 

There is a dubious value of numerical metrics to evaluate the success of such efforts 
(SFRM training) (Musson & Helmreich, 2004). There are many in our own 
organization with the tendency not to believe something unless it can be reduced to 
some form of metric. We were very fortunate in being able to dodge this bullet. It 
allowed us to concentrate our limited resources on refining and delivering the training. 

The single most important lesson and reason responsible for SFRM program 
success at NASA is reinforcement. Our goal was to use team-centered debriefings and 
self-assessment techniques as the primary vehicle for reinforcement of SFRM skills. 
Over time, teams no longer explicitly think of “doing SFRM,” they just end up 
practicing good SFRM and it becomes a natural part of doing business. 


11.17. Concluding Thoughts 


The application of SFRM for astronaut training was a natural progression from the 
commercial and military aviation sectors. However, differences in crew composition to 
include ground-based flight control and management teams added a number of 
challenges. In the course of addressing these challenges, many successes were realized. 
First, SFRM is fully engrained in shuttle flight crew training. Full integration of 
technical and SFRM skill training and reinforcement has been achieved. SFRM 
team-centered debriefs and self-critiquing are now simply “the way things are done.” 
Shuttle instructors realize these are the methods that assure crews “gel” into 
well-coordinated teams. Second, our goal of expanding the SFRM training to the flight 
control community has been successful. During integrated training debriefs (flight crew 
and mission control) SFRM issues are routinely discussed and resolved which has 
resulted in a measured increase in coordination and synergy. Third, being able to bring 
SFRM to management has made profound improvements in the MMT. The 
introduction of team-centered debriefs and individual team member self-assessments has 
clearly been the reason for the team’s continual improvement. SFRM’s most important 
contribution has been the increased health of the organizational safety culture at NASA. 
Looking ahead there is confidence that SFRM will continue with vigor within the 
astronaut and flight control communities. However, there is great concern how SFRM 
training will continue within NASA management. With the eventual end of the Space 
Shuttle Program and emergence of the Constellation Program, it is uncertain how much 
of the progress will transfer. Efforts are currently under way to begin addressing this 
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challenge. SFRM at NASA has been a great success, but achieving exceptional teams 
and sustaining a strong safety culture is not a destination. We must never let our guard 
down and fool ourselves into believing that we have gotten as good as we can get. This is 
a journey—one that we embark on every day to assure the safety of our greatest 
asset—our people. 
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Introduction 


In the aviation industry, Crew Resource Management (CRM) training has become 
the accepted model for developing applied human factors skills among front-line 
employees. In contrast to purely or predominantly knowledge-based human factors 
courses, CRM training typically uses an experiential, adult learning approach to 
provide operational personnel with the understanding of non-technical skills required 
to manage themselves and all available resources safely and effectively. The CRM 
training model, initially developed in aviation, has subsequently been applied 
successfully to a range of other safety critical domains to enhance the performance of 
individuals and teams in both routine and emergency situations. These include the 
maritime and rail industries, healthcare and the offshore oil and gas industry. This 
chapter will examine the migration of the CRM philosophy and training methods to 
these other domains. 


12.1. The Maritime Industry 


The first proliferation of CRM principles beyond the aviation industry occurred in the 
early 1990s, when elements of the international maritime industry became aware of 
the evolution and apparent impact of CRM training within the airline community. 


12.1.1. Rationale for CRM Training in the 
Maritime Industry 


Both maritime and aviation resource management training have their origins in very 
similar occurrences: serious accidents involving the ineffective use of available resources 
on the ship’s bridge or the airliner flight deck (see Barnett et al., 2004; Helmreich and 
Foushee, 1993; Lauber, 1979, 1987, 1993; National Transportation Safety Board, 1979). 
The need for maritime CRM is highlighted by evidence from investigations into 
maritime accidents that reveal critical inadequacies in the ability of individuals to 
manage both resources and emergency situations effectively (e.g. Marine Accident 
Investigation Branch, 1994, 1996, 1999). In particular, the grounding of the QE2 at 
Martha’s Vineyard, Massachusetts, in 1992 prompted the US NTSB to recommend that 
pilots, masters and bridge personnel be trained in Bridge Resource Management 
(National Transportation Safety Board, 1993). Specifically, the report cited failures by 
the vessel’s pilot and master to exchange critical information and a failure to maintain 
situational awareness as probable causes of the accident. 
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The requirement for maritime CRM was further confirmed by British and American 
studies that identified human error as a causal factor in a large proportion of the shipping 
incidents and accidents examined. A study by the shipping insurance industry in the UK 
found that human error accounted for 90% of collision accidents and 50% of cargo 
damage (UK P&I Club, 1997). The US Coastguard estimated that 70% of shipping 
incidents were due to human error (United States Coastguard, 1995, cited in Barnett 
et al., 2004). These figures fit with broadly accepted views on the contribution of human 
action and/or inaction to the vast majority of industrial safety occurrences (see, for 
example, Hollnagel, 2004; Reason, 1990, 1997, 2008; Reason and Hobbs, 2003). 

The results of these various studies and investigations revealed a notable gap in 
maritime training and an opportunity to improve understanding about error prevention, 
detection and management as a means of promoting safety through human behavior. As 
in other transport industries, training for maritime officers had traditionally focused on 
developing individual technical skills rather than addressing the team management, 
communication and coordination issues that are critical for safe operations. 


12.1.2. Development of Maritime CRM 


In the early 1970s the Warshash Maritime Centre in Southampton, UK, developed 
Bridge Operations and Teamwork simulator-based training for ships’ masters and 
officers employed by large oil companies (Haberley et al., 2001). The course included 
training on passage planning and the importance of the relationship between ships’ 
masters and maritime pilots. This course later evolved into what became known as the 
Bridge Team Management (BTM) course. While differing somewhat in philosophy and 
approach, the BTM course comprised many of the topics contained in CRM courses in 
aviation and other industries (Barnett et al., 2004). 

More widely known throughout the maritime industry today is the concept of 
Bridge Resource Management (BRM) training, which first emerged in the early 1990s. 
In 1992, seven major maritime industry bodies! collaborated with the Scandinavian 
Airlines System (SAS) Flight Academy to establish a global Bridge Resource 
Management training initiative (Deboo, undated; Wahren, 2007). This initiative was 
based on the premise that the CRM knowledge and expertise, already developed and 





"Collaborators with SAS Flight Academy on the initial BRM training initiative 
included the Dutch Maritime Pilots’ Corporation, Finnish Maritime Administration, 
National Maritime Administration Sweden, Norwegian Shipowners’ Association, Silja 
Line, the Swedish Shipowners’ Association and The Swedish Club. 
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becoming embedded within the aviation industry, could be beneficially transferred to 
the maritime sector. 

This assumption proved to be correct. The first BRM course was launched in June 
1993, and in subsequent years BRM training, founded on the principles of aviation 
CRM programs, became well established across the global maritime industry, delivered 
principally by The Swedish Club (a large marine mutual insurer) and a number of 
regional licensees. 

In a parallel development, the Danish company Maersk implemented CRM training 
for maritime crews in 1994. Byrdorf (1998) describes the successful adaptation of aviation 
CRM principles to their commercial shipping environment, citing decreases in incident 
and accident rates and lowering of company insurance premiums as a direct result. 


12.1.3. Aim of Bridge Resource Management 
Training 


BRM, as originally developed by the SAS Flight Academy and their maritime industry 
partners, was defined by Deboo (undated) as “the use and coordination of all the skills, 
knowledge, experience and resources available to the bridge team to accomplish or 
achieve the established goals of safety and efficiency of the passage.” 

The aim of BRM was to minimize the risk of incidents by encouraging safe and 
responsible behavior, and to ensure that sound resource management principles 
underpin everyday maritime operations. BRM aimed to foster positive attitudes 
favoring good personal communication, excellence in leadership and compliance with 
operating procedures. Although BRM training remained focused primarily on senior 
bridge crew, some courses included engine room personnel and shore-based marine 
administrators (Deboo, undated). 


12.1.4. Recent Developments: Maritime Resource 
Management (MRM) Training 


After almost a decade of delivering BRM training in many locations around the world, 
a decision was taken to revise and expand the BRM training program. In 2003 the 
organizations involved in the global delivery of BRM training (The Swedish Club and 
BRM licensees) decided to rebrand the course from BRM to MRM (Maritime 
Resource Management) to more accurately reflect the contents and objectives of the 
recently revised training program. The Swedish Club (TSC) currently oversees the 
delivery of MRM training by about 40 training providers across Europe, Asia, the 


The Migration of Crew Resource Management Training - Chapter 12 321 


Americas and Australia. The target audience for MRM training now includes ships’ 
officers, engineers, maritime pilots and shore-based personnel. The stated objective of 
TSC MRM training is to establish genuine safety cultures in shipping companies with 
the ultimate aim of combating the human errors contributing to accidents at sea. 

More recently, one large Dubai-based international shipping operator commissioned 
the development of a fully customised, in-house Maritime Resource Management 
training program. The particular aims of this program were to include contemporary 
aviation CRM concepts and principles such as Threat and Error Management training 
(TEM), and to extend training beyond the bridge to include all elements of ships’ crews. 
To date, a number of courses have been run, involving bridge, engine room and other 
deck personnel, and including all junior and senior officers and ratings, together with 
some shore-based personnel (Dédale Asia Pacific and Vela International Marine, 2006). 

Ship’s Bridge Simulators are today available in a variety of locations around the world 
and many mariners have participated in multiple BTM and/or BRM/MRM simulator 
training sessions. 


12.1.5. MRM Training Delivery 


While techniques vary, the typical four-day TSC MRM training program includes 
multimedia teaching methods and consists of a series of lectures and workshops 
supported by computer-based training (CBT) modules. The original BRM concept had 
each course participant complete a series of CBT modules, exploring maritime 
accidents and incidents from a BRM perspective, with each module reflecting a different 
aspect of the training. At the end of the course, a role-play exercise simulating the 
pressure of a demanding situation allowed participants to practice new BRM-related 
skills. Current TSC MRM courses also employ CBT and some MRM providers use 
“Group-CBT” to deliver the training. 

The recently developed Vela MRM course (Dédale Asia Pacific and Vela 
International Marine, 2006) employs more traditional CRM training methods, using 
peer facilitated classroom presentations, case studies and skill-development exercises to 
convey MRM training concepts and techniques during a three-day course. 


12.1.6. Competency Standards for Non-Technical 
Skills 


The International Maritime Organization’s (IMO) Seafarer’s Training, Certification and 
Watchkeeping (STCW) Code (International Maritime Organization, 1995) specifies 
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mandatory non-technical skills requirements for senior officers with responsibility for 
passenger safety in emergency situations. This minimum standard of competence 
includes organizing emergency procedures, optimizing the use of resources, controlling 
passengers and other personnel during emergency situations, and maintaining effective 
communications. However, the assessment criteria for this standard are based on 
generalized statements of performance outputs as opposed to specific demonstrable 
behaviors and are thus largely subjective (Barnett et al., 2004). 

Thus, although the IMO recognizes the requirement for non-technical skills in 
resource management, the competency standards and assessment criteria need further 
improvement in order to be on a par with aviation standards (Barnett et al., 2004). In 
particular, the development of specific maritime behavioral markers” would be of 
considerable benefit for the assessment of such skills in the shipping industry. 


12.2. CRM in Healthcare 


Anesthesiologists initiated the first adaptations of CRM philosophy and principles to 
health care in the early 1990s. CRM principles are now quite widely discussed and 
practiced within anesthesia and in recent years have begun to infiltrate other areas of 
healthcare (see Helmreich, 1995, 2000; Pizzi et al., 2001), as described below. 


12.2.1. Rationale for CRM in Healthcare 


The delivery of healthcare occurs in a dynamic, complex environment. This makes it 
a routinely high-risk activity further complicated by the potential for life-threatening 
emergency situations. Effective teamwork and crisis management are essential in many 
areas of healthcare, including emergency care, anesthesiology, intensive care and the 
operating room. As in aviation, these specialities require individual healthcare 
professionals with diverse roles and responsibilities to work as an effective, coordinated 
team in preventing, recognizing and managing adverse events. 

Over the last 20 years there has been increasing interest in analyzing and understanding 
errors made by healthcare workers in handling emergency situations. Studies in the late 
1980s (see DeAnda and Gaba, 1990; Gaba and DeAnda, 1989) examined the responses of 
anesthesiologists of different levels of experience to simulated crisis situations in the 
operating room. Findings suggested that anesthesiologists lacked systematic training in 





* Behavioral markers are short, precise descriptors of observable behaviors, typically 
grouped into higher-level categories representing broader performance areas. 
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non-technical skills for critical situations, and that their training was also deficient in 
important aspects of decision-making and crisis management (Gaba et al., 2001). 

Studies have also indicated that similar “false beliefs” or negative CRM-related beliefs 
and behaviors observed in the aviation domain also exist in healthcare. A study reported 
by Sexton et al. (2000) measured attitudes to teamwork, stress and error in operating 
room personnel using an adapted version of a questionnaire developed for use in aviation, 
the Cockpit Management Attitudes Questionnaire (CMAQ; Helmreich, 1984). Results 
showed that junior operating room staff rated teamwork as less effective than their senior 
counterparts, and that surgeons displayed “authoritarian attitudes.’ Only about half of 
those sampled agreed that junior team members should question the decisions of senior 
staff. Some 70% of surgeons reported a belief that even when fatigued they were able to 
perform effectively at critical times. As noted by Merritt (1996) and Helmreich and 
Merritt (1998), very similar beliefs regarding “infallibility” can be found in comparable 
surveys of airline pilots across many different national and organizational cultures. 

Observations of deficiencies in training of medical personnel and a greater awareness 
of the need for effective teamwork and communication have led to a growing consensus 
in healthcare of the potential benefits of CRM-style training. 


12.2.2. Anesthesia Crisis Resource Management 
(ACRM) 


CRM was first introduced in the healthcare domain in the form of Anesthesia Crisis 
Resource Management training (ACRM; Howard et al., 1992; Kurrek and Fish, 1996). 
ACRM was developed in response to the fact that 65% to 70% of threats to safety in 
anesthesiology were found to be caused in part by human error, and the fact that 
anesthetists had little practice in managing crises situations (Howard et al., 1992). 

ACRM courses aim to provide trainees with a range of responses to manage critical 
situations including the ability to coordinate effectively as a team and use all available 
resources in a crisis (Howard et al., 1992). 

One now widely used ACRM curriculum was developed by Gaba and his colleagues 
at Stanford University. The program includes three full-day simulation-based courses 
with increasingly advanced training objectives and goals (Gaba et al., 2001). Each 
ACRM course begins with an introduction to or review of conceptual material, 
followed by group exercises in which participants discuss and analyze presented cases of 
anesthetic “mishaps” or critical situations. The main part of the course is the simulator 
training in which participants manage a number of different crisis scenarios. Participants 
typically play one of three roles in each scenario: the primary anesthesiologist, the first 
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responder (with no knowledge of the scenario) who can be called in to help the primary 
anesthesiologist, or an observer. 

Each scenario is followed immediately by an extensive debriefing session in which 
the management of the crisis is critiqued and analyzed. Debriefings follow 
comprehensive guidelines originally developed for debriefing line-oriented simulation 
exercises in aviation settings (McDonnel et al., 1997) and data gathered from trainees 
have shown that debriefing sessions are considered the most important part of the 
ACRM course (Gaba et al., 2001). 


12.2.3. Effectiveness of Anesthesia Crisis 
Resource Management 


ACRM training is conducted at a number of major teaching institutions and is now 
mandatory on an annual basis for anesthesia trainees at several of these. The 
effectiveness of ACRM training at various centers has been evaluated with different 
methods including questionnaires (e.g. Holzman et al., 1995; Howard et al., 1992) 
and structured interviews (Small, 1998). These evaluation studies show that trainees 
have found their ACRM experience to be very positive, and most believe that it 
contributes to their safe practice of anesthesia (Gaba et al., 2001). For example, 
trainees “uniformly felt that the ACRM course was an intense, superior form of 
training related to an important, but inadequately taught, component of anaesthesia 
practice” (Howard et al., 1992). 

In an effort to establish whether it might be feasible to measure the impact of ACRM 
training on performance, Gaba and his colleagues (Gaba et al., 1998) conducted a study 
to measure anesthesiologists’ technical and behavioral management of simulated crisis 
situations. To assess behaviors in crisis management, 12 markers of CRM were adapted 
from sets previously developed for the evaluation of commercial aviation crews 
(Helmreich et al., 1991). The markers included behaviors such as communication, 
leadership and followership, distribution of workload, and “overall CRM performance.” 

The study revealed many factors that complicate the assessment of the 
effectiveness of simulator-based ACRM training, including high inter-rater 
variability, the need for a large number of subjects, and biases involved in simulator- 
based testing of simulation-based learning. The results suggested that measuring 
ACRM performance was possible, albeit challenging, and that any such evaluations 
would be complex and expensive (Gaba et al., 2001). 

Nonetheless, simulation training for anesthesia trainees based on CRM principles 
has developed rapidly in recent years and is expected to become routine in many other 
healthcare settings in the future (Gaba et al., 2001). 
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12.2.4. Other Adaptations of CRM in Healthcare 


The ACRM approach has been adapted to a variety of other healthcare areas with 
a requirement for effective team performance, including emergency and trauma 
medicine, intensive care and cardiac arrest response teams. An overview of some of 
the approaches used is provided below. 


Emergency team coordination course 


The MedTeams Emergency Team Coordination Course (ETCC) was adapted from 
ACRM and CRM for emergency department care and delivery units (Risser et al., 
1999). The ETCC curriculum covers the five main areas of team structure and climate, 
problem solving, enhancing communication, workload management and team building. 
Like error management training in aviation CRM, the ETCC approach is based on 
improving team performance by avoiding errors, trapping them as they occur and 
mitigating the consequences of errors that do occur (Morey et al., 2002; Shapiro et al., 
2004). 


NeoSim 


ACRM has recently been adapted as the basis of a one-day neonatal resuscitation 
training course for neonatologists and pediatricians called “NeoSim” (Halamek et al., 
2000). The NeoSim course is delivered via didactic instruction with simulation and aims 
to teach behavioral teamwork skills along with some technical content. The simulated 
delivery room places participants in realistic, dynamic crisis situations requiring the 
application of effective technical and behavioral skills in a coordinated team response. 
The training provides delivery room personnel with the opportunity to practice the 
skills necessary to better prepare for, and effectively respond to, crisis situations. 


Team Resource Management 


Team Resource Management (TRM) and Operating Room Crisis Training are two 
versions of CRM customized for the operating room. These courses aim to improve the 
teamwork and error management skills of operating room personnel and focus on topics 
including authority, leadership, communication, decision-making, situation awareness 
and workload management. 


Recent Developments 


An interesting adaptation of CRM to the healthcare domain was implemented recently 
at the Geneva University Hospital in Switzerland (Haller et al., 2008). The objective of 
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this project was to assess the effect of a CRM intervention designed to improve 
teamwork and communication skills in a multidisciplinary obstetric setting. 

The results of the project were evaluated using a pre- and post-training cross- 
sectional study designed to assess participants’ satisfaction, learning and behavioral 
change according to Kirkpatrick’s four-level training evaluation framework: reactions, 
learning, behavior and organizational impact (Kirkpatrick, 1976, 1994). The project was 
conducted in the labor and delivery units of a large university-affiliated hospital, and 
participants included 239 midwives, nurses, physicians and technicians. 

Following completion of the CRM training program the results indicated that 
most participants valued the experience highly, there was significant participant 
learning, and there was a positive change in the team and safety climate in the hospital. 
The authors concluded that implementing CRM training in the multidisciplinary 
obstetrics setting was well accepted by participants and contributed to a significant 
improvement in inter-professional teamwork (Haller et al., 2008). 


12.2.5. The Use of Behavioural Markers in 
Healthcare 


A literature review conducted by Fletcher et al. (2000) indicated that non-technical 
skills checklists and behavioral marker systems are being used quite widely in 
healthcare and that some of these are being used for training, most notably for 
ACRM. Some of these marker systems, including the Operating Room Checklist 
(Helmreich et al., 1995), are actually checklists developed to guide performance and 
actions during training. 

Fletcher and her colleagues (Fletcher et al., 2003) reviewed a number of behavioral 
marker systems and checklists used in anesthesia, and went on to develop a taxonomy of 
behavioral markers for anesthesia based on the existing marker systems and extensive 
consultation with subject matter experts (Fletcher et al., 2004). The resulting markers 
are published in the form of the Anaesthetists’ Non-technical Skills (ANTS) Handbook 
(University of Aberdeen and Scottish Clinical Simulation Centre, 2004). The markers 
cover four main areas important to effective performance in anesthesia: teamworking, 
task management, decision-making and situational awareness. 


12.3. The Rail Industry 


While CRM principles began to appear in isolated outposts of the rail industry more 
than 15 years ago, the international rail community has been comparatively slow to 
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formally adopt applied human factors training. Only in the past five years have any 
coordinated efforts been made to adapt CRM principles to rail operations. 


12.3.1. Rationale for CRM in the Rail Industry 


Rail safety workers face the same challenges as front-line operators in other high-risk 
industries—to ensure safety in a dynamic, demanding operational environment by 
managing threats and errors effectively. Just as other industries have recognized the need 
for specialized training to complement comprehensive technical knowledge and skills, in 
recent years the rail industry in various parts of the world has started to appreciate that 
CRM skills in the areas of communication, coordination, situational awareness, 
decision-making and threat and error management are essential in preventing accidents 
and incidents. 

Accident investigation and research provide strong support for the view that CRM 
training can be of potential benefit for the rail industry. For example, the US Federal 
Railroad Administration (FRA; Federal Railroad Administration, 2002) reports that 
since 1985, human factors issues have accounted for approximately one third of all rail 
accidents and half of all rail yard accidents in the USA. More specifically, human error 
has been indicated as a causal factor in up to 37% of all train accidents not related to 
highway rail grade (level) crossings (Federal Railroad Administration, 1999). Further, 
ineffective CRM-related behaviors have been identified as a contributing factor in 
a number of major rail accidents (e.g. National Transportation Safety Board, 1999a, 
1999b; Office of Transport Safety Investigation, 2004; Transportation Safety Board, 
1998), confirming a significant link between CRM behaviors and safety within the 
industry. 

The NTSB investigation report into a 1998 train collision in the US state of 
Indiana (National Transportation Safety Board, 1999b) concluded that railroad safety 
would be enhanced if rail safety workers received “Train Crew Resource 
Management” training (TCRM) and recommended that such training be developed 
for all train crewmembers. The recommendation stipulated that TCRM training 
should at a minimum address: crewmember proficiency, situational awareness, effective 
communication and teamwork and strategies for appropriately challenging and questioning 
authority. 

The report of a Special Commission of Inquiry into a prominent fatal Australian rail 
accident at Waterfall, NSW, in January 2003 included the recommendation that “Train 
driver and guard training should encourage teamwork and discourage authority 
gradients” (McInerney, 2005a). A subsequent review of safety management systems 
within the rail operator involved in the Waterfall accident was even more prescriptive, 
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recommending “customised human factors training for rail safety workers and 
management/supervisory level staff based on contemporary Crew Resource 
Management principles” (McInerney, 2005b). 

The need for CRM training within the Australian rail industry had been 
identified prior to the Waterfall accident. The investigation of an August 2002 
collision between a passenger train and a derailed ballast train near Bargo, NSW, 
identified deficiencies in post-accident communication and emergency management, 
and attributed these to “inadequate resource management” (Transport NSW, 2002). 
The resulting investigation report included a recommendation that “all Rail Safety 
Workers undertake Crew Resource Management training to increase their 
competence in the use of all resources.” 


12.3.2. Development of CRM in the Rail Industry 


Despite a decade passing since the NTSB’s 1999 recommendation for the introduction 
of Train Crew Resource Management training, research and observation suggests that 
the adaptation and implementation of CRM principles to the rail industry is still in its 
infancy (Dédale Asia Pacific, 2006; Morgan, 2005; Morgan et al., 2003). Some CRM- 
related activities have been undertaken in parts of Europe, for example training in 
communication skills and teamwork in the UK (Mills, 2003; Rail Safety and Standards 
Board, 2004). Until 2007, however, CRM had only been formally adapted to significant 
components of the rail industry in North America, where interest in CRM training 
principles remains sporadic. 

For example, the NTSB reported that in the mid-1990s they became aware of 
a CRM program implemented by the former Southern Pacific Railroad (now Union 
Pacific) that was apparently based on the training provided to flight crew at American 
Airlines (National Transportation Safety Board, 1999b). It is believed that this 
program was established in the late 1980s (Federal Railroad Administration, 2004b). 
While the NTSB reported that Union Pacific has required all new employees to 
undertake this training since 1998, it is difficult to obtain any further details on the 
actual activities being conducted. 

Canadian Pacific Railway (CPR) has conducted a two-day CRM training program 
oriented at new-hire conductors and trainmen since 1999 (Ackerman, 2005). In 2000 
the FRA Railroad Safety Advisory Committee reported that a combined project 
between the AAR and Canadian Pacific had begun developing a generic CRM program 
based on existing CPR materials that could be customized for each individual railroad 
(Federal Railroad Administration, 2000). Deliverables from this program, however, 
could not be located. 
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In 2003, the Texas Transportation Institute (TTI), based at Texas A&M University, 
conducted a study to collect information about the extent of CRM activities in the 
North American rail industry (Morgan et al., 2003). Following an extensive 
consultation process with a cross-section of railroads, Morgan and colleagues reported 
the following: 


m Seven out of ten railroads evaluated had formal CRM programs in place 
in 2003. 


mw Those railroads with active CRM programs generally limited CRM training to 
engineers and/or conductors, although one also provided CRM training to 
dispatchers. 


m At several of the rail companies, CRM training was included in initial training for 
new employees, with less emphasis on a broader program to address recurrent 
training or training for current employees. 


mw While some railroads had no formal CRM program, they did have specific 
programs that taught topics related to key elements of CRM such as situational 
awareness, communication and/or teamwork. 


m Most training programs were classroom based, with material delivered via Power- 
Point and lecture, using videos, group exercises and role-plays to consolidate 
learning. 


m The length of CRM training courses varied from a half-day to two full days, with 
the training sometimes presented in four to five segments over a four- to six-week 
period. 


m Refresher training frequently did not entail additional classroom instruction, 
but was computer based or accomplished through supervisor “ride-alongs” 
which allow a supervisor an opportunity to give specific CRM behavioral 


feedback. 


m CRM program content tended to include the broad topics of situational 
awareness, teamwork, communication and technical proficiency, as well as 
information on human error, safety culture, avoiding distractions, planning, 
fatigue management, assertiveness, briefings, conflict resolution and task 
prioritization. 


The Texas Transportation Institute, working with the FRA’s Office of Railroad 
Development, Office of Safety, and Burlington Northern Santa Fe Railway (BNSF), 
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used this research as the basis for development of a pilot CRM training program for the 
American rail industry (Federal Railroad Administration, 2004a). 


12.3.3. Current Adaptations of CRM in the 
Rail Industry 


Canadian Pacific Railway 


As detailed above, since 1999 Canadian Pacific Railway have conducted a two-day 
CRM training program oriented at their new-hire conductors and trainmen. The 
course was developed by a local community college in Calgary. It is run during the final 
week of a 13-week training program for entry-level employees and focuses on human 
error, teamwork and communication. The course emphasizes the importance of 
teamwork, communication and briefings as countermeasures against human error 
(Ackerman, 2005). 

A working group of rail representatives (including representatives from the 
Association of American Railroads and Norfolk and Southern Railroad) developed 
a short video based on Canadian Pacific’s CRM materials that other railroads then 
customized with their own logos. A number of these railroads subsequently used 
the 30-45 minute training video in safety-related training. 


Federal Railroad Administration 


The FRA railroad industry task force created a generic CRM program for train and 
engine employees. The main topics covered in the program include decision- 
making, assertiveness, crew coordination, leadership, teamwork, situational 
awareness, and active practice and feedback (Federal Railroad Administration, 
2004a). In June 2000, this CRM program was made available to the railroad 
industry. The course syllabus contains ten lesson plans with a coordinating videotape 
that provides opportunities for role-playing, discussion of textbook examples, 
classroom-style instruction and opportunities for group participation. The program 
has three phases: awareness, practice and feedback, and reinforcement (Federal 
Railroad Administration, 2004a). 


Texas Transportation Institute 


Following their evaluation of the status of CRM activities in the North American rail 
industry (see above), TTI has developed a pilot CRM training course which has been 
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tested at several sites on the BNSF Railway (Morgan, 2005). The introductory pilot 
course has three “tracks”: 


m transportation (locomotive engineers, conductors, dispatchers, etc.) 
. ; : . . . 3 
m engineering (maintenance-of-way, signaling, electrical catenary workers,” etc.) and 


m maintenance (locomotive and rolling stock service technicians, mechanical shop 
workers, in-yard train inspectors, etc.). 


The same CRM materials are used for all three “tracks” but the scenarios used as 
examples throughout the course (taken from NTSB or FRA Fatality Reports) are varied 
dependent on the “track” in order to be operationally relevant to the trainees in each 
class. Along with presentation materials, detailed scenario handbooks for trainees and 
scripted instructor guides have been developed for each “track.” 

TTI have subsequently been involved in further research and development of rail 
CRM training practices and materials (Morgan et al., 2006; Olsen, 2005). They 
have also contributed to the development of a business case which demonstrates the 
potential monetary value of CRM training in contributing to safety enhancement 
within the North American rail industry (Federal Railroad Administration, 2007; 
Roop et al., 2007). 


12.3.4. Recent Developments in the Australian 
Rail Industry 


The Australian National Rail Resource Management 
Project 


In 2005 the NSW Independent Transport Safety and Reliability Regulator (ITSRR), 
and Public Transport Safety Victoria (PTSV), with the endorsement of the Rail Safety 
Regulators Panel of Australia and New Zealand (RSRP), initiated a national project to 
develop generic applied human factors training guidelines and materials for the 
Australian rail industry. A consultant was appointed and ITSRR and PTSV, in 
partnership with the rail industry, embarked on a project aimed to improve awareness 
and understanding of the potential benefits of applied human factors training and 





> The catenary is the system of wires suspended above the rail track, which supply 
power to electrically powered trains. 
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provide the rail industry with practical guidance and resources to assist with the 
implementation of such training. 

The resulting reports and materials formed the core deliverables of the National Rail 
Resource Management Project (Dédale Asia Pacific, 2006; Lowe et al., 2007), which 
were completed and launched in December 2007 (Klampfer et al., 2007). Since that 
time, the project sponsors have been actively encouraging the rail industry to introduce 
RRM in some form, drawing on the guidelines and training materials provided. While 
uptake has been slow, a number of rail transport operators have used the materials 
selectively to enhance and supplement pre-existing operational and safety and human 
factors training programs. 

The RSRP has been further involved in part-funding an RRM pilot program to be 
conducted by a major Australian rail operator (Klampfer et al., 2009). In March and 
April 2009, this operator conducted three RRM pilot courses to test the concept and 
some customized training materials. An interesting feature of the training strategy was 
that mixed groups of rail safety workers attended each course (including train drivers, 
conductors, train controllers, signalers, shunters and station staff). Although the training 
was well received by the participants, the overall success and impact of the pilot program 
is being further evaluated. 

The National RRM Project generated significant interest in RRM and human 
factors within the Australian rail industry and also within Europe and the United 
Kingdom. The size and scope of the project inevitably led to many insights and 
opportunities to improve the implementation and integration of human factors 
programs and principles into the rail industry (Klampfer et al., 2009). 


Queensland Rail’s Confidential Observations of 
Rail Safety (CORS) Program 


Queensland Rail (QR) Passenger Services have conducted introductory human factors 
training for train drivers since 2003, using a mix of PowerPoint presentations, case 
studies and facilitated discussions to cover typical human factors topics in a half-day 
course. The latest iteration of the course includes scenario-based simulator practice and 
feedback and a focus on TEM using the results of a rail version of aviation’s Line 
Operations Safety Audit (LOSA). McDonald et al. (2006) provide detail on QR’s 
Confidential Observations of Rail Safety (CORS) program. 


12.3.5. Recent Developments in the UK 


The Rail Safety and Standards Board have recently been investigating the potential 
utility of implementing RRM training to the UK rail industry. A briefing document has 
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been produced which concludes that “the delivery of Rail Resource Management 
training in the GB rail industry has the potential to improve the skills of operational staff, 
reducing the number and severity of accidents and incidents, and bringing financial 
benefits” (Rail Safety and Standards Board, 2009). 


12.4. Offshore Industry 
12.4.1. Rationale for CRM in the Offshore Industry 


The offshore oil and gas industry has a strong teamwork culture and different crews, 
shifts and groups working together typically manage day-to-day operations. Studies of 
accidents and incidents in the offshore industry have found that, as in other high-risk 
industries, human error is frequently identified as a contributing factor (e.g. Flin et al., 
1996; Mearns et al., 1997). 

Closer examination of incidents with human factors causes has demonstrated that 
a high proportion of these are the result of a failure of specific aspects of CRM (Mearns 
et al., 1997). When incidents with human factors causes over a two-year period at seven 
offshore companies were coded into human factors categories, it was found that 46% fell 
within one of the broad CRM topics of teamwork, leadership, situation awareness, 
decision-making, communication, or personal limitations. 

With human error and CRM-related failures apparent as frequent contributors to 
safety events in the offshore environment, it is to be expected that CRM training could 
lead to long-term improvements in the safety performance of the industry. 


12.4.2. Adaptations of CRM to the 
Offshore Industry 


CRM for Offshore Control Room Operators 


CRM was first introduced to the offshore industry when it was adapted for offshore 
control room operators in 1992 (Flin, 1997; Flin and O’Connor, 2001). Elements of 
CRM from the aviation domain were incorporated into a training program focusing on 
emergency response training and competence assessment for offshore control room 
operators. 

The course covered four standard CRM topics adapted for control room 
operations: decision-making, communication, stress and assertiveness. Modules used 
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by commercial airlines were modified to suit the specific needs of the domain by 
drawing on psychological research and the expertise of control room operations 
trainers. 

Course delivery included lectures as well as exercises and discussion of personal 
experiences relating to the topic areas. Four principal modules were developed: decision- 
making, communication, assertiveness and stress. 


CRM for Offshore Installation Managers and Emergency 
Response Teams 


CRM has also been developed for Offshore Installation Managers (OIMs) and their 
teams undergoing emergency response training in control room simulators (Flin, 1997; 
O’Connor and Flin, 2003). The course was designed around elements that were critical 
for effective team performance in emergency command centers, including 
understanding of team roles, communication, group decision-making, assertiveness, 
team attitudes, stress management and shared mental models. The aim of the course was 
to improve team performance in the emergency control center. 


Emergency Resource Management—Elf Norge 


CRM training has also been used with Norwegian offshore crews, in the form of 
Emergency Resource Management training at Elf Petroleum Norge (Grinde, 1994; Flin 
et al., 2000). The overall aim of the course is to provide operators with a comprehensive 
understanding of the resources available during an emergency. The course includes 
elements of decision-making, task allocation, situation assessment and communication. 
The initial three-day course is delivered by lectures, supported by practical scenarios run 
in an onshore simulator. 


CRM for Offshore Production Installation Crews 


The first prototype CRM program for offshore production installation crews was run in 
1999 (Flin et al., 2000; O’Connor and Flin, 2003). This two-day course was designed to 
improve safety and productivity for production and maintenance crews, as well as to 
reduce down time for these two groups. Course content included modules on fatigue 
and shiftwork, stress, team coordination, communication, decision-making, situation 
awareness, and an understanding of human error and the origins of CRM. Evaluation of 
the eight courses completed suggested that this type of CRM training could have benefit 
for the offshore oil and gas industry. 
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12.4.3. The Use of Behavioral Markers 


The prototype CRM for offshore production installation crews described above aimed 
to provide participants with an increased awareness of a set of non-technical skills. The 
non-technical skills framework was based on research and the NOTECHS framework 
developed for aviation (Flin et al., 2003). Specific objectives were written for each 
module around the non-technical skills framework in the early stages of course 
development (Flin et al., 2000). 


12.5. Future Migration 


As discussed above, the philosophy and principles of CRM training have been successfully 
adapted to a number of other safety-critical work domains. Those not discussed in 
detail here include the Nuclear Power Production industry (e.g. Belton, 2001; Gaddy and 
Wachtel, 1992) and also the US Space Program (e.g. see Rogers, 2002). 

So, what does the future hold for resource management training outside the aviation 
domain? While CRM has migrated successfully from the aviation flight deck to the 
bridges and engine rooms of merchant vessels, hospital emergency departments and 
operating theaters, offshore oil platforms, and more recently rail transport systems, the 
adoption of such training initiatives has not been universal, and permanence is far from 
guaranteed. Where CRM in aviation has evolved considerably over the past 30 years, 
with airline operators frequently leading the way, followed by regulators who have in 
many cases enshrined CRM training principles in regulatory requirements and 
guidelines (e.g. Civil Aviation Authority, 2003; Federal Aviation Administration, 2004), 
this is not yet the case in the other domains discussed above. In the absence of greater 
industry-wide acceptance, regulation and guidance, while training principles and 
techniques may evolve over time, the majority of shipping companies, hospitals, rail 
operators and petroleum companies may choose to do without this important applied 
human factors training. 


REFERENCES 


Ackerman, F, 2005. CRM training at Canadian Pacific Railway. Personal Commu- 
nication December 2005. 

Barnett, M., Gatfield, D., Pekcan, C., 2004. A research agenda in Maritime Crew 
Resource Management. Paper presented at the National Maritime Museum 
Conference on Safer Ships, Safer Lives, London, March 2004. 


336 Chapter 12 - The Migration of Crew Resource Management Training 


Belton, S., 2001. CRM training in the nuclear industry. Paper presented at the 
Third CRM Users Group Workshop, University of Aberdeen, October 2001. 
Cited in Flin, R., O’Connor, P., Mearns, K., 2002. Crew resource management: 
improving safety in high reliability industries. Team Performance Management 8, 
68-78. 

Byrdorf, P., 1998. Human factors and crew resource management: an example of 
successfully applying the experience from CRM programmes in the aviation world 
to the maritime world. Paper presented at the 23rd Conference of the European 
Association for Aviation Psychology, Vienna, September 1998. 

Civil Aviation Authority, 2003. CAP 737: Crew Resource Management (CRM) 
Training. Guidance for Flight Crew, CRM Instructors and CRM Instructor- 
Examiners. CAA Safety Regulation Group, London. 

DeAnda, A., Gaba, D., 1990. Unplanned incidents during comprehensive anesthesia 
simulation. Anesthesia and Analgesia 71 (1), 77—82. 

Deboo, K.N. (undated). Maritime Resource Management. Anglo-Eastern Maritime 
Training Centre, Mumbai. 

Dédale Asia Pacific, 2006. Interim Report, National Rail Resource Management 
Project: Review of Best Practice, Implementation Issues and Task Analysis. PTSV/ 
ITSRR, Melbourne/Sydney. 

Dédale Asia Pacific, Vela International Marine 2006. Maritime Resource Management 
(MRM) Training Course, Author, Melbourne. 

Federal Aviation Administration, 2004. Crew Resource Management Training. 
AC120-51E. US Department of Transportation, Author, Washington, DC. 

Federal Aviation Administration, 2005. Line Operations Safety Audit (LOSA). Draft 
Advisory Circular. US Department of Transportation, Author, Washington, DC. 

Federal Railroad Administration, 1999. Railroad Safety Statistics Annual Report. 
United States Department of Transportation, Office of Public Affairs, Author, 
Washington, DC. 

Federal Railroad Administration, 2000. Minutes of the Railroad Safety Advisory 
Committee Meeting, 28 January 2000. United States Department of Transportation, 
Office of Public Affairs, Author, Washington, DC. 

Federal Railroad Administration, 2002. Five-Year Strategic Plan for Railroad Research, 
Development, and Demonstrations. United States Department of Transportation, 
Office of Public Affairs, Author, Washington, DC. 

Federal Railroad Administration, 2004a. Safety Assurance and Compliance 
Program (SACP). Year 2003 Accomplishments. US Department of Trans- 
portation Federal Railroad Administration. Office of Safety, June 2004. Author, 
Washington, DC. 


The Migration of Crew Resource Management Training - Chapter 12 337 


Federal Railroad Administration, 2004b. Switching operations fatality. Findings 
and recommendations of the SOFA working group. August 2004 update. 
United States Department of Transportation, Office of Public Affairs, Author, 
Washington, DC. 

Federal Railroad Administration, 2007. Rail Crew Resource Management (CRM): 
The Business Case for CRM Training in the Railroad Industry (DOT/FRA/ 
ORD-07/21). United States Department of Transportation, Office of Public Affairs, 
Author, Washington, DC. 

Fletcher, G., Flin, R., McGeorge, P. 2000. WP1 Report: Review of Human Factors 
Research in Anaesthesia. Version 1. Interim Report on SCPMDE Research Grant 
RDNES/991C/. 

Fletcher, G., Flin, R., McGeorge, P. 2003. WP2 Report: Review of Behavioural 
Marker Systems in Anaesthesia. Version 1.1. Work package 2 Report on SCPMDE 
Research Grant RDNES/991/C. 

Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N., Patey, R., 2004. Rating 
non-technical skills: developing a behavioural marker system for use in anaesthesia. 
Cognition, Technology and Work 6, 165—171. 

Flin, R., 1997. Crew Resource Management for teams in the offshore oil industry. 
Team Performance Management 3 (2), 121—129. 

Flin, R., Mearns, K., Fleming, M., Gordon, R., 1996. Risk Perception and Safety in 
the Offshore Oil and Gas Industry. Report (OTH 94454). HSE Books, Suffolk. 
Flin, R., O’Connor, P, 2001. Crew Resource Management in the offshore oil 
industry. In: Salas, E., Bowers, C., Edens, E. (Eds.), Improving Teamwork in 

Organizations. LEA, New Jersey. 

Flin, R., O’Connor, P., Mearns, K., Gordon, R., Whitaker, S., 2000. Factoring the 
Human into Safety: Translating Research into Practice. vol. 3—Crew Resource 
Management Training for Offshore Operations. OTO 2000 063. HSE Books, Sudbury. 

Gaba, D.M., DeAnda, A., 1989. The response of anesthesia trainees to simulated 
critical incidents. Anesthesia and Analgesia 68 (4), 444—451. 

Gaba, D.M., Howard, S.K., Fish, K.J., Smith, B.E., Sowb, Y.A., 2001. Simulation- 
based training in anesthesia crisis resource management (ACRM): a decade of 
experience. Simulation and Gaming 32 (2), 175—193. 

Gaba, D., Howard, S., Flanagan, B., Smith, B., Fish, K., Botney, R., 1998. Assessment 
of clinical performance during simulated crises using both technical and behavioral 
ratings. Anesthesiology 89 (3), 8—18. 

Gaddy, C., Wachtel, J., 1992. Team skills training in nuclear power plant operations. 
In: Swezey, R., Salas, E. (Eds.), Teams: Their Training and Performance. Ablex, 
New Jersey. 


338 Chapter 12 - The Migration of Crew Resource Management Training 


Grinde, T.A., 1994. Emergency Resource Management training. In Proceedings of the 
Second International Conference on Health, Safety, and the Environment in Oil and 
Gas Exploration and Production. Richardson, Texas: Society of Petroleum Engineers 
vol. 2, Jakarta, Indonesia. pp. 413-417. 

Haberley, J.S., Barnett, M.L., Gatfield, D., Musselwhite, C., McNeil, G., 2001. 
Simulator training for handling escalated emergencies. MCA Project RP 467. 
Warshash Maritime Centre, Southampton. 

Halamek, L.P., Kaegi, D.M., Gaba, D.M., Sowb, Y.A., Smith, B.C., Smith, B.E., et al., 
2000. Time for a new paradigm in pediatric medical education: teaching neonatal 
resuscitation in a simulated delivery room environment. Pediatrics 106, E45. 

Haller, G., Garnerin, P., Morales, M.-A., Pfister, R., Berner, M., Irion, O., Clergue, E, 
Kern, K., 2008. Effect of crew resource management training in a multidisciplinary 
obstetrical setting. International Journal for Quality in Health Care, Advance Access 
published May 6, 2008. 

Helmreich, R.L., 1984. Cockpit management attitudes. Human Factors 26, 583—589. 

Helmreich, R.L., 1995. Interpersonal human factors in the operating theater. Paper 
presented at the Danish Anaesthesia Simulator Conference, Copenhagen. 

Helmreich, R.L., 2000. On error management: lessons from aviation. British Medical 
Journal 320, 781-785. 

Helmreich, R.L., Butler, R.E., Taggart, W.R., Wilhelm, J.A., 1995. Behavioural 
markers in accidents and incidents: reference list (NASA/UT/FAA Technical Report 
95-1). The University of Texas, Austin, TX. 

Helmreich, R.L., Foushee, H.C., 1993. Why crew resource management? Empirical 
and theoretical bases of human factors training in aviation. In: Wiener, E.L., 
Kanki, B.G., Helmreich, R.L. (Eds.), Cockpit Resource Management. Academic 
Press, San Diego, CA, pp. 3—45. 

Helmreich, R.L., Merritt, A.C., 1998. Culture at Work in Aviation and 
Medicine: National, Organisational and Professional Influences. Ashgate, Aldershot, 
UK. 

Helmreich, R., Wilhelm, J., Kello, J., Taggart, W., Butler, R., 1991. Reinforcing and 
evaluating crew resource management: evaluator/LOS instructor reference manual 
(NASA/UT Technical Manual 90-2, Revision 1). NASA/University of Texas 
Aerospace Crew Performance Project, Austin. 

Hollnagel, E., 2004. Barriers and Accident Prevention. Ashgate, Aldershot, UK. 

Holzman, R.S., Cooper, J.B., Gaba, D.M., Philip, J.H., Small, S.D., Feinstein, D., 
1995. Anesthesia crisis resource management: real-life simulation training in oper- 
ating room crises. Journal of Clinical Anesthesia 7 (8), 675—687. 


The Migration of Crew Resource Management Training » Chapter 12 


Howard, S.K., Gaba, D.M., Fish, K.J., Yang, G., Sarnquist, FH., 1992. Anesthesia crisis 
resource management training: teaching anesthesiologists to handle critical incidents. 
Aviation Space and Environmental Medicine 63, 763—770. 

International Maritime Organization, 1995. Seafarers Training, Certification and 
Watchkeeping Code (STCW Code). IMO, London. 

Kirkpatrick, D.L., 1976. Evaluation of training. In: Craig, R.L. (Ed.), Training 
and Development Handbook: A Guide to Human Resources Development. 
McGraw-Hill, New York. 

Kirkpatrick, D.L., 1994. Evaluating Training Programs: The Four Levels. Berrett- 
Koehler, San Francisco, CA. 

Klampfer, B., Grey, E., Lowe, A., Hayward, B., Branford, K., 2009. Reaping the 
benefits: how railways can build on lessons learned from crew resource management. 
Proceedings of the Third International Conference on Rail Human Factors. Lille, 
France. March 2009. 

Klampfer, B., Walsh, C., Quinn, M., Hayward, B., Pelecanos, S. 2007. The national rail 
resource management (RRM) project. Launch presentation, Sydney, December 
2007. 

Kurrek, M.M., Fish, K.J., 1996. Anesthesia crisis resource management training: an 
intimidating concept, a rewarding experience. Canadian Journal of Anesthesia 43, 
430—434. 

Lauber, J.K., 1979. Resource management on the flight deck: background and state- 
ment of the problem. In: Cooper, G.E., White, M.D., Lauber, J.K. (Eds.), Resource 
Management on the Flight Deck: Proceedings of a NASA/Industry Workshop, 
San Francisco, June 1979 (NASA Conference Publication 2120). NASA Ames 
Research Center, Moffet Field, CA. 

Lauber, J.K., 1987. Cockpit resource management: background and overview. In: 
Orlady, H.W., Foushee, H.C. (Eds.), Cockpit Resource Management Training: 
Proceedings of the NASA/MAC Workshop (NASA Conference Publication 
CP-2455). NASA Ames Research Center, Moffet Field, CA, pp. 5-14. 

Lauber, J.K., 1993. Foreword. In: Wiener, E.L., Kanki, B.G., Helmreich, R.L. (Eds.), 
Cockpit Resource Management. Academic Press, San Diego, CA, pp. xv—xviii. 
Lowe, A.R., Hayward, B.J., Dalton, A.L., 2007. Guidelines for rail resource 
management. Report prepared by Dédale Asia Pacific for Public Transport Safety 
Victoria and Independent ‘Transport Safety and Reliability Regulator, 

NSW. Melbourne/Sydney: PTSV/ITSRR. 

Marine Accident Investigation Branch, 1994. Report of the Chief Inspector of Marine 
Accidents into the Engine Failure and Subsequent Grounding of the Motor Tanker 
Braer. MAIB, Southampton. 


339 


340 Chapter 12 - The Migration of Crew Resource Management Training 


Marine Accident Investigation Branch, 1996. Report of the Investigation into the 
Power Failure on Canberra. MAIB, Southampton. 

Marine Accident Investigation Branch, 1999. Marine Accident Report 5/99, 
Report of the Inspector’s Inquiry into the Loss of MV Green Lily. MAIB, 
Southampton. 

McDonald, A., Garrigan, B., Kanse, L., 2006. Confidential observations of rail safety: 
an adaptation of Line Operations Safety Audit (LOSA). Paper presented at the 
Multimodal Symposium on Safety Management and Human Factors. Swinburne 
University, Australia. February 2006. 

McDonnell, L., Jobe, K., Dismukes, R., 1997. Facilitating LOS debriefings: a training 
manual (NASA Technical Memorandum 112192). NASA Ames Research Center, 
Moffett Field, CA. 

McInerney, P.A., 2005a. Special Commission of Inquiry into the Waterfall Rail 
Accident. Final Report, Vol. 1. Sydney: NSW Government. 

McInerney, P.A., 2005b. Special Commission of Inquiry into the Waterfall Rail 
Accident. Final Report, vol. 2. Sydney: NSW Government. 

Mearns, K., Flin, R., Fleming, M., Gordon, R., 1997. Human and Organisational 
Factors in Offshore Safety (OTH 543). HSE Books, Suffolk, UK. 

Merritt, A.C., 1996. National culture and work attitudes in commercial aviation: 
a cross-cultural investigation. The University of Texas at Austin: Unpublished 
doctoral dissertation. 

Mills, A., 2003. The growth of human factors as a discipline in the UK rail industry. 
Paper presented at the Fifth Australian Aviation Psychology Symposium. Australia, 
Sydney. December 1—5, 2003. 

Morey, J.C., Simon, R., Jay, G.D., Wears, R.L., Salisbury, M., Dukes, K.A., et al., 
2002. Error reduction and performance improvement in the emergency department 
through formal team work training: evaluation results of the MedTeams project. 
Health Services Research 37, 1553-1581. 

Morgan, C.A., 2005. Texas Transportation Institute CRM pilot project. Personal 
communication, December 2005. 

Morgan, C.A., Kyte, T.B., Olson, L.E., Roop, S.S., 2003. Assessment of Existing 
Teams and Crew Resource Management (CRM) Training within the Rail Industry. 
Texas Transportation Institute. November 15, 2003. Presented at Transportation 
Research Board 2004 Annual Meeting. 

Morgan, C., Olson, L.E., Kyte, T.B., Roop, S., Carlisle, T.D., 2006. Railroad Crew 
Resource Management (CRM): survey of teams in the railroad operating environ- 
ment and identification of available CRM training methods. Report produced by 


The Migration of Crew Resource Management Training »- Chapter 12 341 


Texas Transportation Institute for the US Department of Transportation, Federal 
Railroad Administration. 

National Transportation Safety Board, 1979. Aircraft Accident Report. United 
Airlines, Inc., DC-8-61, N8082U, Portland, Oregon, December 28, 1978 (Report 
No. NTSB AAR-79-7). Author, Washington, DC. 

National Transportation Safety Board, 1993. The grounding of the UK passenger 
vessel RMS Queen Elizabeth 2 near Cuttyhunk Island, Vineyard Sound Massa- 
chusetts, August 7th 1992. NTSB Report Number: MAR-93-01. Author, 
Washington, DC. 

National Transportation Safety Board, 1999a. Railroad Accident Report. Collision 
between Union Pacific Freight Trains MKSNP-01 and ZSEME-29 near Delia, 
Kansas. July 2, 1997 (Report No NTSB/RAR-99/04). Author, Washington, DC. 

National Transportation Safety Board, 1999b. Railroad Accident Report. Collision of 
Norfolk Southern Corporation Train 255L5 with Consolidated Rail Corporation 
Train TV 220 in Butler, Indiana, on March 25, 1998 (Report No. NTSB/RAR-99/ 
02). Author, Washington, DC. 

O’Connor, P., Flin, R., 2003. Crew resource management training for offshore oil 
production teams. Safety Science 41, 111-129. 

Office of Transport Safety Investigation, 2004. Rail Safety Investigation Report. 
Unanderra. Signal passed at danger resulting in derailment of Pacific National 
Service B9162, 20 June 2003. Reference number 00041. Sydney. 

Olsen, L.E., 2005. CRM training in rail. Personal communication, December 
2005. 

Pizzi, L., Goldfarb, N., Nash, D., 2001. Crew Resource Management and its appli- 
cations in medicine. In: Making Health Care Safer: A Critical Analysis of Patient 
Safety Practices. Evidence Report/Technology Assessment, No. 43. Chapter 44CA. 
Stanford University Evidence-based Practice Center, University of California at San 
Francisco. 

Rail Safety and Standards Board, 2004. Teamworking in the Railway Industry. The 
Journey Guide, v1.1. London. 

Rail Safety and Standards Board, 2009. Rail resource management training: a guide for 
the UK rail industry. RSSB Briefing Document, May 2009. London. 

Reason, J., 1990. Human Error. Cambridge University Press, Cambridge. 

Reason, J., 1997. Managing the Risk of Organisational Accidents. Ashgate, Aldershot, 
UK. 

Reason, J., 2008. The Human Contribution: Unsafe Acts, Accidents and Heroic 
Recoveries. Ashgate, Aldershot, UK. 


342 Chapter 12 - The Migration of Crew Resource Management Training 


Reason, J., Hobbs, A., 2003. Managing Maintenance Error. A Practical Guide. 
Ashgate, Aldershot, UK. 

Risser, D.T., Rice, M.M., Salisbury, M.L., Simon, R., Jay, G.D., Berns, S.D., 1999. 
The potential for improved teamwork to reduce medical errors in the emergency 
department. The MedTeams Research Consortium. Annals of Emergency Medicine 
34, 373-383. 

Rogers, D.G., 2002. NASA’s Space Flight Resource Management Program: 
a successful human performance error management program. Paper presented at 
Space Ops Conference, 2002. 

Roop, S.S., Morgan, C.A., Kyte, T.B., Arthur, Jr., W., Villado, A.J., Beneigh, T., 2007. 
Rail crew resource management (CRM): the business case for CRM training in the 
railroad industry. Report produced by Texas Transportation Institute for the US 
Department of Transportation, Federal Railroad Administration. 

Sexton, J.B., Thomas, E.J., Helmreich, R.L., 2000. Error, stress, and teamwork in 
medicine and aviation: cross sectional surveys. British Medical Journal 320, 745-749. 

Shapiro, M.J., Morey, J.C., Small, S.D., Langford, V., Kaylor, C.J., Jagminas, L., 
Suner, S., Salisbury, M.L., Simon, R., Jay, G.D., 2004. Simulation based teamwork 
training for emergency department staff: does it improve clinical team performance 
when added to an existing didactic teamwork curriculum? Quality and Safety in 
Health Care 13 (6), 417—421. 

Small, S., 1998. What participants learn from anesthesia crisis resource management 
training. Anesthesiology 89 (3A), A71. 

Transport, NSW, 2002. Bargo-Yerrinbool Derailment and Collision, 1 August 2002. 
Final report. Author, Sydney. 

Transportation Safety Board, 1998. Railway Investigation Report. Rear-end Train 
Collision, 11 August, 1998 (Report number R98V0148). Quebec. 

UK P&I Club, 1997. Analysis of Major Claims—Ten-year Trends in Maritime Risk. 
Thomas Miller P&I Ltd, London. 

United States Coastguard, 1995. Prevention through People, Quality Action Team 
Report. USCG, Washington, DC. 

University of Aberdeen and Scottish Clinical Simulation Centre, 2004. Anaesthetists’ 
Non-Technical Skills Handbook, v1.0. Aberdeen. 

Wahren, E., 2007. Development of BRM training at SAS Flight Academy. Personal 
communication, June 2007. 


PART 3 





CRM Perspectives 


Chapter 13 


A Regulatory 
Perspective 


Kathy H. Abbott 
United States Federal Aviation Administration’ 





' The views represented in this chapter are those of the author and do not represent 
an official position of the Federal Aviation Administration. 





Crew Resource Management 
Copyright © 2010, by Elsevier Inc. All rights of reproduction in any form reserved. 


345 


346 Chapter 13 - A Regulatory Perspective 


Introduction 


This chapter presents Crew Resource Management (CRM), primarily from the 
perspective of the United States (US) regulatory system, administered by the Federal 
Aviation Administration (FAA). The chapter begins with a brief primer of the under- 
lying philosophy of aviation regulation as a form of risk management. It then describes 
the history and philosophical basis of some of the key aviation regulations in the US 
system. 

The chapter will then focus on specific aspects of CRM: crew coordination and 
communication, error management and flight crew monitoring. The chapter will then 
discuss how these aspects of CRM fit into the regulatory structure for equipment design, 
flight crew training and flight crew procedures. The chapter will present a discussion of 
future examples where implementation of CRM is important and will conclude with 
a description of where regulatory material for resource management is applied to other 
areas besides flight crews. 


13.1. Aviation Regulation — A Brief Primer? 


One way to look at regulation is to consider it as a form of risk management, and to 
consider where a society places certain activities on a notional continuum of risk. 
Figure 13.1 shows such a continuum, which describes private risk at one end and public 
risk at the other. It depicts different activities and where they may be placed by a society. 
In the continuum shown in the figure, the left end of the continuum represents private 
risk, with activities such as scuba diving, mountain climbing and skiing as examples of 
items that a society might place on this end. These activities at this end correspond to 
significant personal choice and freedom, with low public concern and responsibility. 

Activities considered to be of high public concern and responsibility are represented 
on the right end of the continuum, and examples include commercial flights and 
national defense. In the USA (and many other societies), large aircraft accidents are 
placed to the right end of the continuum, because they are considered to be of high 
public concern and responsibility. Highway accidents and small airplane accidents are 
further to the left on the scale, since this is the choice of the society. 





* The material in this section is based on the regulatory primer perspective from the 
RTCA Task Force on Certification (RTCA, 1999). In particular, the contributions 
of several members of RTCA Task Force 4 on Certification were instrumental in 
developing the material for the regulatory primer, including John Ackland, Tony 
Broderick and Tom Imrich. 
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Figure 13.1 “Personal” vs. “public” risk assumption 
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Another society may choose to place these activities and safety-related events at 
different points on the continuum. For example, another society may choose to place 
private aviation at the same “risk” point (and therefore the same level of public concern) 
as commercial aviation. 

Society also determines the role of the government in managing or mitigating risk. 
Consider the continuum shown in Figure 13.2, which illustrates where the potential 
government role may be. The leftmost end is intended to represent activities or concerns 
that are primarily personal and commercial. At this end, the society chooses to limit the 
government’s role—possibly in a role to simply enable the activity. At the rightmost end, 
one can find activities or concerns that are inherently governmental, such as national 
defense. At this end, the government actually conducts or controls the activity. 

The location of any particular activity is driven by the will of the society for which 
the government works. In the USA, for example, private aviation is considered to be less 


Figure 13.2 Continuum depicting potential governmental role 
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of a public concern, and therefore less of a public responsibility by the government. 
Commercial aviation is much more a public concern, and is expected to have the highest 
level of safety. Therefore, legislation requires that government oversight and standards 
are more stringent for commercial aviation than for private aviation. 

US aviation regulations were initiated through the Air Commerce Act of 1926, 
which assigned responsibility and regulatory authority for aviation to the 
Aeronautics Branch of the Department of Commerce. This branch had the following 
objectives: 


1. 


2. 


a: 


establish airworthiness standards and associated system of aircraft registration 


administer examination and licensing procedures for aviation personnel and 
facilities 


establish uniform rules for air navigation 


. establish new airports and 


encourage the development of civil aviation. 


Basically, the fundamental governmental responsibilities are to assure: 


1. 


2. 


that aircraft don’t fall on the public (assure the airworthiness of the aircraft) 
the “highest level of safety” for public transportation 
at least a basic level of safety for other “certificated aircraft” passengers and 


that aircraft can satisfy safety-related inter-aircraft responsibilities for mutual 
separation (e.g. the requirement for altitude-encoding transponders in certain 
airspace). 


The means used to accomplish these responsibilities are: 


1. 
2. 
3. 


certifying air vehicles and supporting ground elements—if and as necessary 
establishing operating rules—“rules of the road” and 


providing or empowering certain capabilities (e.g. certain services, facilities, or 
capabilities agreed to by the aviation system users, or by the public). 


In part, these functions are accomplished via some type of “certification.” Here, 
certification means the approval and authorization for aircraft; personnel (e.g. pilots); 
operations; procedures; facilities; and equipment. 
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The legal origin for the FAA’s regulatory activities is founded in the US Constitution 
and is generally considered to have begun with the Air Commerce Act, enacted in 1926. 
This act commissioned the Secretary of the Department of Commerce is responsible for 
fostering air commerce, issuing and enforcing air traffic rules, certifying pilots and 
aircraft, and operating and maintaining air navigation aids (NAVAID). Birnbach and 
Longridge (1993) provided both a historical perspective on the FAA and a more detailed 
history of the evolution of the legal structure. 

The current regulations that the FAA administers are contained in the US Code, 
specifically Title 14—Code of Federal Regulations (CFR), Aeronautics and Space 
Chapter I—FAA, Department of Transportation. Three subchapters of particular 
interest to this discussion are Subchapter C, Aircraft, Subchapter F Air Traffic and 
General Operating Rules, and Subchapter G, Air Carriers and Operators for 
Compensation or Hire: Certification and Operations. Subchapter C includes the 
Airworthiness Standards for various categories of aircraft (including Part 25 for 
Transport Category Airplanes). Subchapter F contains the general operating and flight 
rules (Part 91) and Subchapter G contains Part 121 Operating Requirements: Domestic, 
Flag and Supplemental Operations, and Part 135— Operating Requirements: 
Commuter and On-Demand Operations and Rules Governing Persons on Board Such 
Aircraft. 

Part 25 and several other parts in Subchapter C contain airworthiness standards for 
aircraft. These requirements are considered to be point in time regulations, because once 
compliance is found with one of these regulations (such as issuance of a Type Certificate 
for an airplane type design) it is not revisited unless the type design of the airplane 
changes (e.g. adding equipment or systems not part of the original type design). 
Therefore, any change in any of the regulations in Part 25 does not result in the change 
of existing certificated aircraft type designs. 

In contrast, the operating rules (Parts 91, 121, 135, etc.) are continuous applicability 
rules, and therefore, when a regulation is changed, the operators certificated under that 
operating rule must comply according to the date the rule is effective. Crew training 
requirements fall under the operating regulations, allowing changes to be made 
continuously as our understanding of good practices in such training improves. 

The discussion above describes the underlying philosophy of the regulations, which 
represents requirements. But regulatory material can have one or more motivations, as 
discussed below: 


m Minimum standards. The regulations might describe the minimum standards 
for a required characteristic, such as the performance of a system on an 
aircraft. 
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mw Protection, such as 14 CFR Part 193. This part describes when and how the 
FAA protects from disclosure safety and security information that is submitted 
voluntarily to the FAA. 


m Incentives for equipage by giving operational credit. For example, aircraft with 
autoland capability (and corresponding pilot qualification) have the potential to 
fly to lower visibilities than aircraft without such capability. 


The FAA publishes several other types of documents, in addition to regulations. 
One such type of document is an advisory circular (AC), which provides guidance 
from the FAA to the external community. An AC may contain guidance on 
means of compliance with particular regulations, or may provide other information 
of interest to the aviation community (for example, one AC lists all the 
published ACs). 

Advisory circulars are numbered using a system that corresponds to the regulations 
for which it provides information. For example, AC 25.1329 Approval of Flight 
Guidance Systems (FAA, 2006b) provides a means of compliance with 14 CFR 
25.1329 (FAA, 2006a). Another example of an AC is AC 120-76A for Approval of 
Electronic Flight Bags (FAA, 2003b). This includes both airworthiness and operational 
approval guidance for approving a technology or type of system, rather than a specific 
regulation. 

The material below discusses both regulations and advisory circulars related to 
CRM, in airworthiness of equipment design, flight crew training and operational 
approval requirements and guidance. 


13.2. Regulatory Requirements and Guidance 
for Crew Resource Management—Flight Deck 


The following important aspects of CRM (among others) are addressed in the 
regulatory material and will be described in the following sections: 


1. Crew coordination and communication 
2. Error management and 
3. Flight crew monitoring. 


For each of these aspects, this chapter will describe how it is addressed in regulatory 
material for equipment design, flight crew training and flight crew procedures. 
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13.2.1. Airworthiness Requirements for 
Equipment Design—Examples 


Two regulations will be discussed below to illustrate how considerations for CRM are 
incorporated into the airworthiness requirements for equipment design: 


1. 14 CFR Part 25 Section 25.1329 Flight Guidance Systems 


This regulation describes the airworthiness requirements for Flight Guidance Systems 
(FGS), including autopilots, autothrust systems, flight directors and associated flight 
crew interfaces (FAA, 2006a). Operational experience showed that flight crew errors 
and confusion were occurring when operating the FGS and its subsystems (FAA, 1996), 
including vulnerabilities that can be mitigated in the equipment design. Therefore, 
the airworthiness requirements were updated to address these issues, and to address 
changes in technology and capabilities of FGS. 

As one example of how the equipment design requirements were updated to support 
crew coordination, paragraph (j) requires that the alert for autopilot disengagement must 
be done in a way to assure that the information is available to each pilot: 


(j) Following disengagement of the autopilot, a warning (visual and auditory) must be 
provided to each pilot and be timely and distinct from all other cockpit warnings. 
(FAA, 2006a) 


The Advisory Circular for this regulation makes it clear that the intent is that the 
alert associated with disengagement of the autopilot(s) must be implemented in a way to 
support flight crew coordination: 


It should sound long enough to ensure that it is heard and recognized by the pilot and other 
flight crewmembers, but not so long that it adversely affects communication between 


crewmembers or is a distraction. 
(FAA, 2006b, p. 25) 


Paragraph (i) explicitly addresses the need to support error management through 
preventing errors, and through the equipment design providing feedback on current 
modes of operation: 


(i) The flight guidance system functions, controls, indications, and alerts must be designed to 
minimize flightcrew errors and confusion concerning the behavior and operation of the flight 


guidance system. 
(FAA, 2006a) 
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Paragraph (i) also says: “Means must be provided to indicate the current mode of 
operation, including any armed modes, transitions, and reversions. Selector switch 
position is not an acceptable means of indication. The controls and indications must be 
grouped and presented in a logical and consistent manner. The indications must be 
visible to each pilot under all expected lighting conditions” (FAA, 2006a). This portion 
of the regulation supports the requirement for equipment design to support crew 
monitoring of the status of the FGS. 


2. European Aviation Safety Agency (EASA) Certification 
Specification 25.1302 Installed Systems for use by the Flight 
Crew (EASA, 2007 a) 


Another airworthiness regulation was developed jointly by the FAA, the Joint Aviation 
Authorities (JAA), the European Aviation Safety Agency (EASA), North and South 
American industry, and European industry to address the need for the equipment design 
to support error management by the pilots. This regulation was written to require the 
equipment design to have characteristics that are known to avoid error. Specifically, 
the equipment must provide the information and controls necessary for the pilots to do 
the tasks associated with the intended function of the equipment, and the controls 
and information must be in a usable form. 

In addition, the regulation was written based on the understanding that even 
well-qualified pilots using well-designed systems will make errors. Therefore, the 
equipment design must support detection and recovery aspects of error management. 
The first sentence of paragraph (d) explicitly addresses this: 


(d) To the extent practicable, installed equipment must enable the flight crew to manage errors 
resulting from the kinds of flight crew interactions with the equipment that can be reasonably 
expected in service, assuming the flight crew is acting in good faith. 

(EASA, 2007a)° 


With respect to crew monitoring, CS 25.1302 requires that the equipment design 
provide the information needed to perform the tasks associated with the intended 
function of the equipment—and this includes monitoring of the equipment—and that 
the equipment provide information about its operationally relevant behavior. The FAA 
plans to harmonize with the requirements in CS 25.1302, which will result in the USA 
and Europe having consistent requirements for this aspect of equipment design. 





> As with all regulations, the regulatory material should be read and considered in its 
entirety, together with other applicable regulations. 
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The airworthiness regulations described above are “point in time” regulations, as 
discussed earlier. Therefore, any new airplane type must meet the requirements, but 
existing airplanes do not. 

Since there are many aircraft that received their aircraft certification approval 
before these regulations were implemented, such aircraft do not necessarily meet the 
requirements for the equipment design to support crew coordination, error 
management and crew monitoring. Thus, the mitigations required in the flight crew 
training and procedures are especially important for such aircraft. 


13.2.2. Flight Crew Training Requirements 


The US regulations include a requirement for training of CRM principles and topics for 
pilots and dispatchers. These requirements for CRM are codified into 14 CFR Part 121 
Section 121.404, Compliance dates: Crew and dispatcher resource management 
training, which states: 


After March 19, 1998, no certificate holder may use a person as a flight crewmember, and 
after March 19, 1999, no certificate holder may use a person as a flight attendant or aircraft 
dispatcher unless that person has completed approved crew resource management (CRM) or 
dispatcher resource management (DRM) initial training, as applicable, with that certificate 
holder or with another certificate holder. 

(FAA, 1996) 


The requirement extends beyond air carriers operating under Part 121. On 
December 20, 1995, the FAA published Air Carrier and Commercial Operator Training 
Programs. This final rule required CRM training for crewmembers in their training 
programs by certificate holders conducting operations under Part 135 that are required 
to comply with Part 121 training and qualification requirements, such as those certificate 
holders that conduct commuter operations with airplanes for which two pilots are 
required by aircraft certification rules, and those that conduct commuter operations with 
airplanes having a passenger seating configuration of ten seats or more. A Notice of 
Proposed Rulemaking published on May 1, 2009, would require all certificate holders 
conducting operations under Part 135 to include CRM training in their programs. It 
would continue the precedent set by the December 20, 1995 final rule. 

The regulation itself does not specify the content of the training, but Advisory 
Circular (AC) 120-51E (FAA, 2003b) provides guidance for the content of US operator 
training programs to address CRM. Subjects such as crew coordination and 
communication, error management and flight crew monitoring are specifically 
described in the AC. This AC also discusses the importance of pre- and post-training 
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session briefings, and ways to evaluate the pilots’ performance as a result of the training, 
among other topics. As an example of related guidance outside the USA, CAA UK 
(2006) provides guidance for the content of CRM training. 

And as with all the operating regulations, the requirement for CRM training for 
flight crews is a continuous applicability requirement. Thus, improvements to the 
guidance can be made and applied as more is learned about effective implementation of 
training for CRM. 


13.2.3. Flight Crew Procedures 


The FAA recognizes that flight crews should use procedures that embody the 
coordination and communication intended by CRM. The design of procedures should 
embody that coordination. Degani and Weiner (1994) describe that there are several 
aspects to design of the procedures that can promote crew coordination: 


1. Reduced variance. The procedure triggers a predetermined and expected set of 
actions. 


2. Feedback. Procedures specify expected feedback to other crew members (e.g. 
callouts). This feedback can detail (1) the current, and/or expected system state; 
(2) the actions that are currently being conducted; (3) the system outcome; and 
(4) an indication of task completion. There are several ways in which this 
feedback is provided: (1) verbally (callouts, callback, etc.); (2) nonverbally 
(gestures, manual operation—such as pulling down the gear lever); (3) via the 
interface (when the configuration of the system is significantly changed, e.g., 
all displays are momentarily blank when power is switched from APU to 
engine-driven generators, this provides clear feedback to the other pilot); and 
(4) via the operating environment (when slats/flaps are extended during 
approach, there is a clear aerodynamic feedback—pitch change). 


3. Information transfer. Procedures convey, or transfer, information from one agent 
to others (e.g. the after takeoff checklist is complete). 


Another area that is important to consider for crew coordination is the delineation of 
duties among the pilot flying (PF), the pilot not flying (PNF)/pilot monitoring (PM) 
and the flight engineer (if present). This includes the identification of who does which 
tasks, which pilot calls for particular procedures, which pilot reads them and which one 
responds. 

The FAA has published an advisory circular that provides guidance for 
implementation of Standard Operating Procedures (SOPs) that address these areas, 
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including recommended templates (FAA, 2003a). This AC points out that effective crew 
coordination and communication depend on the crew’s having a shared mental model of 
each task, and that mental model is founded on SOPs. The procedures’ templates 
included in the AC provide recommended steps for crew coordination and 
communication tasks. 

This AC specifically highlights the need for pilots to perform monitoring tasks, 
based on recognition of the role that inadequate monitoring played in previous 
accidents. For example, the National Transportation Safety Board has identified that 
inadequate crew monitoring or challenging was involved in 84% of crew-involved 
accidents (NTSB, 1994; Sumwalt, 2004). 

The AC also discusses the conversion of the term “pilot not flying” to “pilot 
monitoring” to reflect what the pilot is doing, rather than what the pilot is not doing. In 
addition, it helps to emphasize the importance of the monitoring task. This emphasis 
must be part of the philosophy that forms the foundation of the SOPs. 

An example that illustrates the performance improvement that can result from 
modifications to procedures to support crew coordination is the Altitude Awareness 
Programs implemented by several airlines (Sumwalt, 1995). In this example, airlines 
were experiencing altitude deviations, or altitude “busts” —cases where the pilots were 
not leveling off at the cleared altitude or were going to different altitudes than the one to 
which they were cleared. The formal Altitude Awareness Program implemented at these 
airlines was based on recognition that it is essential that a crew cross-check each other, 
and challenge each other when there is a doubt about the air traffic clearance. Key 
elements of the procedures for changing the altitude in the flight deck include setting the 
altitude alerter and making callouts. 

This program provides an example where delineating pilot duties is very important. 
The successful program at US Air, with reduction in altitude busts of approximately 
75%, as compared to preprogram figures (Sumwalt, 1995), illustrates that significant 
improvement in performance can result from appropriate crew coordination and 
epia delineation of duties, with crew monitoring as a key task among those 
duties. 


13.3. Future Considerations 


Aviation has always been about change. As civil aviation moves to improved future 
airspace operations, successful implementation of new technology and new operational 





* See Sumwalt (1995) for detailed discussion of the characteristics of the procedures and 
for lessons learned from operational experience. 
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concepts will need to include the basic aspects of CRM discussed above—coordination 
and communication (among all the participants); error management; and monitoring. 
The discussion below will highlight two areas where these aspects must be 
implemented for safe, effective and efficient operations. These two areas are: Electronic 
Flight Bags (EFBs) and the integration of the flight deck operations as part of airspace 
operations. 

Although laptop computers have been in use for many years in the flight deck, as 
computer technology gets smaller, more powerful and more affordable, use in the 
flight deck is expanding—as are the variety of applications for which they are being 
used. These devices can range from installed systems to portable, handheld systems. 
The classes of device are described in AC 120-76A (FAA, 2003b) and Joint Aviation 
Authorities (AA) Temporary Guidance Leaflet No. 36 (JAA, 2004). These devices 
are being used for an increasing variety of applications, including such tasks as 
performance calculations, moving map displays for surface operations and many 
others. 

Several sources have identified human factors issues (including CRM, among many 
others) related to EFBs and their use (see Allen, 2003 and Chandra et al., 2003 for 
detailed discussions). Accordingly, the approval guidance from the FAA and JAA/EASA 
identify assessment of CRM as part of the training and system evaluation (FAA, 2003b; 
JAA, 2004). This may be especially important because use of EFBs can change the flight 
crew’s way of interacting and communicating, especially if the EFB is located such that 
cross-flight deck viewing is difficult. If the pilots cannot see each other’s EFBs, then 
explicit coordination and communication must be done to mitigate the lack of visibility 
into each other’s actions. In addition, crew procedures must be carefully defined to 
include error management (including, specifically, cross verification of data entry and 
computed information). 

The use of EFBs in the flight deck is still evolving, and it will be important to 
continue to address the role of CRM in this use—and in the integration of EFBs with 
use of paper. According to Nomura et al. (2006): 


The complex, high-stakes, high-tempo nature of the pilots’ work requires careful planning of 
information access and the management of attention. The fact that shared understandings are 
essential to safe flight means that whatever the representations are, they must not only be 
available to both pilots, but available to the pilots jointly in interaction with one another. 
While engaging in a briefing preceding a high-workload maneuver such as a takeoff, pilots 
want to locate themselves bodily in an environment that is rich in tangible representations of 
the parameters relevant to upcoming events. Currently they do this by reading across many 
disparate documents and displays. A better understanding of how this is done could 
contribute to the next generation of display design. 
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According to this work, paper is an integral feature of using new technologies and 
plays important social interaction roles in crew coordination, message confirmation, 
note-taking and information affordance. 

This is just one example that illustrates that the introduction of new technologies into 
the flight deck must consider crew coordination and communication, and the support of 
pilot tasks and interactions. Training and flight crew procedure design often do consider 
these aspects of CRM, but the equipment design must support these tasks as well. 

The second example expands consideration of the CRM concepts to the interaction 
between the flight crew and the air traffic personnel, as a key part of integration of 
aircraft operations in the airspace. Communication and coordination between pilots and 
air traffic services personnel has been done successfully in operations for many years. 
However, new airspace operations are expected to make significant changes in 
operations, with corresponding benefit. 

One change that is under way right now is the move towards a performance-based 
navigation system in both the USA and around the world. Performance-based 
navigation incorporates the use of Area Navigation (RNAV) equipment that is not 
reliant on the location of ground-based navigation aids. In addition to the point-to- 
point capabilities offered by RNAV, new procedures are also being implemented 
that incorporate Required Navigation Performance (RNP) (Barhydt and Adams, 2006). 

RNAV and RNP procedures offer significant benefits to both operators and air 
traffic managers. These benefits include better access to terrain-limited airports, more 
environmentally friendly flight paths and significant gains in airspace efficiency. 
Performance-based operations are being implemented in both the terminal area and en 
route environments. The implementation of these procedures has already produced 
tangible benefits at a number of different airports. 

One of the benefits of new RNAV procedures in the terminal area is reduced 
air/ground communications (Barhydt and Adams, 2006). But the reduction in 
communications raises potential concerns about the quality of the communication and 
coordination between air and ground, and the management of errors—preventing, 
detecting and correcting them. Therefore, the CRM concepts should be applied within 
the flight deck, between air traffic personnel, and between the pilots and air traffic 
services. As new regulatory and policy material is developed to enable these new 
operations, such material should specifically include the CRM concepts. 


13.4. Concluding Remarks 


This chapter has presented a regulatory perspective on CRM, given primarily from a US 
point of view. It is not comprehensive, since CRM considerations can be found 
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throughout the regulatory material related to flight deck airworthiness, flight crew 
training and flight crew procedures—among other application areas. Nor are the 
examples for future operations a comprehensive set. Rather, they are just intended to 
illustrate examples of future needs. 

The FAA and other regulatory authorities around the world recognize the 
importance of CRM as a necessary and significant contributor to aviation safety. 
Although most of this chapter has focused on the pilots, the regulatory system is 
recognizing the importance of resource management for other personnel in the 
aviation system, including dispatchers (as evidenced by the requirement for 
dispatcher resource management training), stated above, and guidance for training 
of maintenance personnel (FAA, 2000). 

This propagation of resource management considerations throughout the 
regulatory material reflects the growing understanding of the importance of this 
area—but it needs to be even more widespread. Application of CRM concepts to 
air traffic personnel, to the communication and coordination between pilots and 
ATS, among pilots, maintenance, dispatchers, cabin crew, and others remain 
important and can be improved even further. 
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Introduction 


This chapter takes the regulatory perspective from the view of the Voluntary Safety 
Programs Branch at FAA Headquarters. As a policy office, the Voluntary Safety 
Programs Branch sponsors Crew Resource Management (CRM) research, meets 
annually with all of the researchers, reviews airline training and checking scenarios, fields 
questions from airline and FAA field personnel and reviews CRM assessment strategies 
of all air carriers subject to the policies of the branch. The branch’s programs provide 
a major source of CRM data to the airlines and the FAA. The Advanced Qualification 
Program (AQP) is the most advanced delivery system for training and measuring CRM; 
thus this office is the FAA OPR (Office of Primary Responsibility) for three of the most 
important CRM-related Advisory Circulars (Federal Aviation Administration 2004a, 
Federal Aviation Administration 2004b, Federal Aviation Administration 2006a), and it 
has sponsored most of the FAA-funded crew resource management research programs 
for air transport category aircraft pilots over the last 20 years. 


14.1. Historical Perspective 


The history of the FAA’s evolving understanding of and guidance for CRM may be 
traced through the evolution of several of its key CRM-related Advisory Circulars: AC 
120-35 Line Oriented Flight Training Programs, AC 120-51 Cockpit Resource 
Management and AC 120-54 Advanced Qualification Program. Tracing the historical 
arc of these guidance materials shows a progression from more abstract concepts to more 
concrete examples and recommendations, based on airline experience with the 
programs. The Line Oriented Flight Training (LOFT) Advisory Circular (120-35) is 
used as an illustration. 

The initial guidance on LOFT (120-35 and 35A) preceded the guidance on CRM, 
and was relatively conceptual, but the document was issued because FAA wanted to 
recognize and endorse the concept, encouraging airlines to experiment with the 
approach. On the one hand, the original primary aim of LOFT was not so much CRM 
skills, as those skills were understood at that time, but the rehearsal of operational 
procedures in an operational setting (Butler, 1993). On the other hand, one of the goals 
of LOFT was to expose flight crews to an error chain in a non-jeopardy environment to 
see how they would manage it, which is entirely consistent with contemporary 
understandings of CRM. 

LOFT subsequently morphed into Line Operational Simulation (LOS), which later 
became the primary vehicle for training and assessing CRM skills at many airlines. AC 
version 120-35B expanded the concept of LOFT to LOS, to include LOFT, Special 
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Purpose Operational Training (SPOT) and Line Operational Evaluation (LOE). The 
treatment of SPOT and LOE was, again, extremely sparse, with a single page dedicated 
to each. The goal, again, was to recognize and endorse a practice in concept, and 
encourage the airlines to work it into a practical procedure. The current version, 
120-35C, incorporates more procedural guidance based on recommendations developed 
by the Airline Transport Association (ATA) AQP Working Group (ATA, 1995), which 
has come to be known as the event set methodology for scenario development. This 
methodology was subsequently adopted by the FAA as a recommended procedure for 
all LOS development, for both training and assessment. Thus, the FAA followed its 
pattern of recognizing gains and encouraging practical applications. 

The various revisions to the CRM advisory circular followed a similar pattern. The 
FAA was quick to adopt concepts in large part from the research community, but 
conservative in providing procedures or mandates until industry experience could be 
coalesced into a concrete method or process. Initial CRM training would later be 
mandated for 14 CFR Part 121 crews (121.404), followed by a mandate for recurrent 
training (121.427). But these requirements were mandated only after close to a decade of 
FAA guidance that was purely advisory in nature. 


14.1.1. Fighting Old Battles 


In the first edition of this volume, (Wiener, Kanki and Helmreich, 1993), Birnbach 
and Longridge (1993) introduced the Advanced Qualification Program and describe 
its potential impacts on CRM training. They proposed that by measuring CRM at 
the task level, as opposed to measurement at a more global level, it might be feasible 
to measure CRM with levels of validity and reliability that would enable pass/fail 
grading of CRM skills with the same level of confidence check airmen currently 
express about making pass/fail decisions when grading technical skills. They proposed 
a five-year trial period, during which CRM would be assessed on a no-jeopardy 
basis. 

While the idea of pass/fail CRM grading proved to be politically untenable with 
many pilot organizations, the effort to achieve this goal did drive a line of research that 
substantially enhanced the practice of CRM assessment. The FAA agreed that until new 
methodologies could be developed and tested, no evaluation event was to be failed on 
the basis of CRM alone within the AQP training programs. The reasoning was that if 
CRM was extremely poor, it would eventually manifest itself as the failure of a technical 
task. The technical task would be recorded as the primary reason for the failure, while 
the CRM problems that led to the technical shortfall would be documented as 
contributing factors only. This allowed the air carrier community to document CRM 
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shortfalls without failing a pilot on a check purely on the basis of CRM skill deficiencies. 
This compromise was workable for the FAA, the carriers and the pilot’s unions. 


14.2. Measuring and Grading CRM 


The FAA did in fact sponsor this line of research. In cooperation with the ATA AQP 
Working Group and its member airlines, the research community helped develop the 
event set methodology for the design of scenarios for instructing and assessing CRM 
skills (ATA, 1994; Seamster et al., 1995). Researchers worked with major and regional 
carriers to develop processes for inter-rater (Holt et al., 2001) and referent-rater 
reliability (Goldsmith & Johnson, 2002), for the establishment and use of gold standards 
in pilot evaluation (Baker & Dismukes, 2003), recommendations for grading scales and 
the establishment of reasonable benchmarks for their measurement (Williams et al., 
1996; Holt et al., 2002). Although many airlines participated in the research and dozens 
of papers, presentations, special issues (Baker and Dismukes, 2002) and even books 
(Holt, 2001) were generated by the research, in the end most of the airlines and the 
pilots’ organizations did not want to see pass/fail CRM evaluations as part of AQP. The 
FAA decided not to move forward with pass/fail CRM as a requirement, but only as an 
option. To this day, the ALPA website carries the notice that ALPA supports AQP only 
so long as the pass/fail evaluation of CRM is not mandated. It is not. Most participating 
AQP airlines capture numerical grades for technical tasks and then assign associated 
letter designations labeled “reason codes” to capture any CRM factors that contributed 
to that numerical grade. 

At the conclusion of a three-year effort at a regional carrier, the George Mason 
University Research Team prepared a report of their findings for FAA internal 
consumption (GMU, 2001). It included several measures of reliability applied to both 
technical and CRM scoring. This report concludes as follows: 


The basic point that the table makes is that some values of the reliability of measures of 
CRM performance are as good as the reliability values for measuring technical performance. 
For example, the agreement on overall CRM for 5 event sets is .86 for the regional carrier, 
which is as good as the agreement index obtained for the overall technical performance for the 
same 5 event sets of.85. Similarly, the internal consistency estimate for CRM for 11 event 
sets (.75) is slightly higher than for the consistency of overall technical ratings on those event 
sets (.66). These data suggest that we can measure CRM performance with at least the same 
degree of consistency as we can measure technical performance. 


These data provided the FAA with confidence in the toolsets that were now in place 
for developing and assessing event-based scenarios: the grade scales, the event set 
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methodology itself, the techniques for teaching and measuring CRM tasks, and the 
calibration tools and schedules. 

This situation creates and interesting irony for AQP training, as the FAA Practical 
Test Standards (FAA, 2001) clearly allow for pass/fail judgments based solely on CRM 
issues. There is a caveat that warns the inspector or check airman to be very conservative 
in failing a pilot on CRM skills alone, referring to this as an “entirely subjective” 
judgment call, but one that is none-the-less authorized. AQP, with all of the controls it 
has put in place to minimize subjectivity, does not require pass/fail CRM grading. 
Traditional training guidance, without those controls designed to limit subjectivity, 
permits pass/fail grading of CRM. 


14.3. The Role of the Regulator 


This chapter addresses the role of the FAA in the history of the development of CRM 

as falling into three broad categories: the enforcement of minimal regulatory requirements, 
the encouragement of practices that exceed the regulatory minimums and the funding of 
research programs, which in part allow the FAA to determine where to draw the line 
between what will be strictly mandated and what will be simply encouraged. 


14.3.1. Enforcing the Minimums 


The primary vehicle traditionally used by the FAA to maintain safety standards is the 
establishment, oversight and enforcement of mandatory requirements, and the 
publication of those standards as rules, regulations and policy. This guidance is developed 
by specialized policy offices at FAA Headquarters in Washington, DC. The FAA usually 
coordinates these rules with industry representatives and always examines them for 
compliance with ICAO directives. As a member of ICAO, the FAA must file differences 
with ICAO in those cases where FAA rules do not align with ICAO requirements. Once 
rules are issued, field FAA personnel are not permitted to authorize major deviation 
from those rules. Certificate holders must petition for an exemption from the FAA for 
such deviations. 

One exception to this process applies to air carriers training under 14 CFR 121 
Subpart Y: The Advanced Qualification Program. While carriers who train under 
traditional 14 CFR Part 121 and Part 135 regulations must still turn to the exemption 
process for regulatory deviations (alternative means of compliance), AQP carriers do 
not. AQP was designed to allow air carriers to explore more creative and contemporary 
approaches to training and checking. Because of the additional measures and controls 
engineered into the AQP process (front end analysis, job task analysis, student entry 
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analysis, performance measurement using scalar data, inter- and referent-rater reliability, 
performance data analysis, etc.) alternative means of compliance are less risky and 
therefore the process for deviation from regulations is engineered into the process, 
avoiding the need for most training exemptions. Without the inherent flexibility of 
AQP, the air carrier community would not have been able to evolve advanced CRM 
techniques as quickly as it did. Traditional prespective training regulations do not lend 
themselves to supporting innovation the way the AQP regulations do. 

From time to time the FAA has encouraged the industry to voluntarily exceed 
regulatory minimums on their own, but these have traditionally been the exception rather 
than the rule. The introduction of the Aviation Safety Reporting system (ASRS) program 
in 1974 isan example of this type of exception, one of the earliest voluntary safety programs 
sponsored by the FAA. This heavy reliance on mandates and compliance shifted somewhat 
abruptly in the early 1990s, when the FAA formalized five new voluntary safety programs 
in a five-year period (the Advanced Qualification Program and Voluntary Disclosure 
Reporting Program in 1990, the Internal Evaluation Program in 1992 and the Flight 
Operations Quality Assurance and Line Operational Safety Audit Program in 1995). 

By 2002 the Air Transportation Division of the Flight Standards Service found it 
necessary to rename the policy office that handles these programs the Voluntary Safety 
Programs Branch (AFS-230) to recognize and accommodate the unique needs of this 
portfolio of safety programs (FAA, 2002). In spite of this increased focus on voluntary 
programs, the establishment of mandatory minimums remains the principal regulatory 
foundation of aviation safety within the USA as well as internationally. 


14.3.2. Rules 


The rules for training and evaluating pilots, flight engineers, flight attendants and 
dispatchers are contained in 14 CFR Parts 121 and 135, which saw its last major rewrite 
in the early 1970s (a Notice of Public Rulemaking (NPRM) for the next major rewrite 
was published for public comment in January of 2009). As the appreciation for the 
importance for CRM training grew over time, those changes were reflected in changes 
to these rules, first to CFR 14 Part 121, and later through 14 CFR Part 119, which 
required the same training for both CFR 14 Parts 121 and 135. 


14.3.3. Policy 


While rules are developed at a high level of generality in order to provide a stable, long- 
term foundation for safety, the specific means of compliance with those rules is fleshed 
out in more detail in policy. The issuance of a rule is, for example, routinely 
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accompanied by the issuance of an advisory circular for industry to follow, and an entry 
in FAA Order 8900.1, which is guidance to the FAA workforce as to how to provide 
oversight of the subject of the rule and advisory circular. Some programs are based on 
rules, such as AQP and FOQA, while some rest only on policy, such as ASAP and 
LOSA. Not every policy has a rule, but every rule has a policy. 

Today, the primary policy guidance for CRM is found in the current versions of 
three advisory circulars: AC 120-51 Crew Resource Management, AC 120-35 Line 
Operational Simulations and AC 120-54 Advanced Qualification Program. The 
voluntary safety programs office, which is the FAA’s Office of Primary Responsibility 
(OPI) for AC 120-35 and 120-54, has recently become the OPR for 120-51. This will 
permit these three primary guidance documents to be more easily harmonized and 
should help bring an increased level of cohesion to the FAA’s effort to standardize and 
oversee CRM training and checking. For many years the policy office that sponsored 
most of the CRM research for the FAA’s Air Transportation Division was not the same 
policy office responsible for the CRM Advisory Circular. This reassignment of OPR 
responsibilities will rectify that disconnect as well. AC 120-51 is the primary CRM 
policy document for pilots and flight attendants. Dispatch Resource Management 
training is addressed by AC 121-32A (Federal Aviation Administration 2005). 


14.3.4. Rewarding Best Practices 


While statutes, rules, regulations and policy are the bedrock of safety, such mandates 
have their own natural limitations. Not all problems are solved by prescriptive rule- 
making. The more rules the FAA publishes, the more complex, difficult and expensive 
the tasks of both compliance and surveillance become. The unintended consequence of 
promulgating more and more rules and restrictions is that it leads to shortcuts, errors and 
an increase in violations (Reason, 1995). 

The FAA cannot write a rule for every possible contingency, nor are the most 
challenging safety concerns necessarily addressed by prescriptive procedures. Reason 
(1997) dedicates an entire chapter to the various dilemmas and double binds the 
regulators find themselves in when developing guidance. The underspecification of 
guidance increases errors while the overspecification of guidance increases violations. 
The balance between too much and too little guidance is based on complex factors 
whose cause and effect relationships are poorly understood. 

In order to enhance safety the FAA has developed a series of creative voluntary 
programs that add to aviation safety without providing an undue burden of extra rules for 
operators to follow or for the FAA to enforce. These programs provide regulatory 
incentives to operators to collect safety data above and beyond rule compliance, and to act 
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on that safety data to improve their operations. Because the programs are voluntary, carriers 
that find their requirements too burdensome need not participate. These programs have 
developed into a major source of CRM data for both the industry and the FAA. 


14.3.5. Voluntary Programs 


The FAA offers the air carrier community several types of voluntary programs. In some 
cases, early and voluntary compliance with an anticipated new rule is encouraged and 
facilitated by the FAA, but it is not required. This allows the operator to implement and 
experiment with the new requirements before they are mandated. This is a temporarily 
voluntary program. The current Safety Management System (SMS) is an excellent example 
of such a voluntary program that will ultimately become a regulatory requirement in the 
US. The FAA has issued an Advisory Circular (FAA, 2006b) that contains the SMS 
standard and encourages certificate holders to implement the elements of the program on 
a voluntary basis. The FAA has fielded teams who are working through the implementation 
process at numerous certificate holders, even though the final rule is probably several years 
away, and the final implementation deadline as much as five years beyond that. 

Other programs are intended to remain voluntary on a permanent basis. These 
programs are considered to be too complex and expensive to mandate for all certificate 
holders, and are therefore encouraged rather than required. Within the Flight Standards 
Service of the FAA, these programs are managed by a single policy office, the Voluntary 
Safety Programs Branch. The programs fall into three broad categories, to be recognized 
in the future SMS rule: 


Reporting Programs 
Aviation Safety Reporting Program (ASRS) 
Aviation Safety Reporting Program (ASAP) 
Flight Operations Quality Assurance (FOQA) 
Voluntary Disclosure Reporting Program (VDRP) 
Auditing Programs 
Internal Evaluation Program (IEP) 
Line Operations Safety Audits (LOSA) 
Training Program 


Advanced Qualification Program (AQP) 
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Because of the expense and complexity of some of these programs, it is primarily 
the larger carriers that participate in all of them. What they all have in common is that 
they are voluntary in nature, and provide some mechanism for collecting and 
examining data that is then used to identify and act on safety issues. Beyond that, each 
has its own particular features. ASRS is funded by the FAA, managed by NASA, and 
available to everyone in the aviation community. VDRP is funded and managed by the 
FAA, and is available to any certificate holding organization. IEP is simply guidance 
provided to the operator community to assist them in developing an internal auditing 
system. AQP is available to CFR Part 121 and 135 air carriers, as well as CFR Part 
142 training centers. FOQA was developed for all aircraft operators. ASAP was 
developed for CFR Part 121 air carriers and Part 145 repair stations, but has been 
allowed in other CFR Parts on a case-by-case basis. LOSA is available to any flying 
organization. 

What is the role of CRM in these programs? CRM events, issues and performance 
problems are reported to NASA/FAA through ASRS, to the operator/FAA through 
ASAP and AQP, to the operator (and in some cases the FAA) through LOSA and IEP, 
and to a much lesser extent to the operator and FAA through FOQA and to the FAA 
through VDRP. These latter programs are not as rich in CRM findings as are the 
former programs, but CRM issues do surface through follow-on investigations of these 
events. These voluntary programs supplement the traditional data sources, such as 
accident reports, incident reports and captain’s reports. Most of these programs include 
taxonomies to organize and process human factors and crew resource management 
data. 


14.3.6. Regulatory Incentives 


All voluntary programs provide some sort of incentive for operator participation. 
Because all involve soliciting safety data from operators and individuals that would not 
necessarily be available to the FAA without such programs, the certificate holders, 
whether organizations or individuals, need some sort of “carrot” to participate, 
including the confidence that the “carrot” will not turn into a “stick” when the FAA 
receives the information. Hence, most programs deal in de-identified data, reductions 
from legal enforcement actions to no more than administrative actions, or agreements to 
take no action based on voluntarily submitted data, so long as criminal or deliberate acts 
are not committed and corrective actions are successfully implemented. These programs 
have extended the reach of CRM data available not only to the FAA, but to the industry 
as well. The FAA’s new Office of Aviation Analytic Services (ASA-1) has facilitated 
Memoranda of Understanding between its support contractor, MITRE Corporation, 
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and a number of airlines to share safety data through a virtual archive. The program 
began with ASAP and FOQA data, but will continue to add additional databases as it 
expands its national reach. 


14.3.7. Funding Research 


One of the most important contributions the FAA has made to CRM over the last few 
decades has been to fund teams of researchers at universities, research centers and even 
private firms to further the understanding of CRM. While some of this research was 
targeted to the support of specific program requirements, in particular AQP, much of it 
was dedicated to supporting promising efforts to explore the fundamental elements of 
CRM. While some researchers were asked to develop specific toolsets for industry 
application, such as the best methodologies for inter- and rater-referent training and 
evaluation for CRM factors, other researchers were funded to simply explore the 
boundaries of the field and report their findings. 


14.3.8. Human Factors 


By the mid-1980s the FAA, NASA and NTSB all came to the conclusion that a primary 
cause of accidents was essentially the independent, self-sufficient pilot who tried to 
maintain safe flight through independent heroics, instead of relying on the full range of 
resources available to assist in the maintenance of safe flight. These pilots were not 
communicating, coordinating, or employing the wide range of support tools at their 
disposal. This was perceived as a lack of human factors skills, and so the FAA engaged in 
a serious effort to fund a Human Factors Research, Engineering and Development 
(RE&D) program. While most FAA RE&D had to this time been traditionally funded 
and conducted at the FAA Technical Center in Atlantic City or the Mike Monroney 
Aeronautical Center in Oklahoma City, this Human Factors RE&D was to be funded 
out of FAA Headquarters in Washington, DC and sponsored by the various policy 
offices located there. 


14.3.9. Crew Resource Management 


For the last 20 or so years, the FAA Office of the Chief Scientist for Human Factors has 
directed an air carrier training research program centering on developing methods for 
effective pilot training and assessment. Central to this work was the study of Crew 
Resource Management (CRM) and its many intricacies. While considerable CRM 
work was conducted prior to the birth of AQP, the AQP office was the first to take full 
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advantage of the ability to sponsor this type of research. Based on the needs of the 
sponsoring organization, the real impetus for this particular line of research was the AQP 
mandate that CRM be integrated into all aspects of pilot training and that CRM skills be 
assessed. Although basic CRM concepts had been widely accepted by industry by the 
introduction of AQP in 1990, much remained to be learned regarding the appropriate 
methods for effective training and valid and reliable assessment of training programs as 
called for in AQP. 

The general research philosophy guiding efforts to improve CRM training and 
assessment was that research must consider the distinct segments of aviation training 
systems. Individuals comprising the crew, instructors who train and evaluate crews in the 
classroom, the simulator and on the line, as well as the management personnel 
responsible for in the safety climate of the carriers should all be considered. Additionally, 
this research had to consider how CRM was implemented and assessed in Line Oriented 
Flight Training (LOFT) and Line Operational Evaluation (LOE). Thus, this research 
centered on (1) crew training and assessment, (2) instructor training and (3) LOFT and 
LOE development strategies. 

This program included many talented researchers who have contributed substantial, 
vital information and training and assessment tools to both the FAA and air carrier 
industry. Some of these researchers, such as Robert Helmreich, Judith Orasanu, Barbara 
Kanki and Eduardo Salas, have authored chapters for this volume, so their work will not 
be addressed here. The following represents only a subset of the organizations and studies 
that made up the program. 


14.4. Evaluator Calibration 


One of the first challenges for this line of research was to devise methods for maximizing 
the validity and reliability of CRM assessment (Holt and Johnson, 1997). One of the first 
studies tried to improve assessment by designing new rater training and data collection 
systems. This work focused on rater calibration, development of grade sheets, 
development of videos used in the calibration sessions, presentations to the instructors 
on the importance of quality data, collection and analysis of the calibration data, and 
formats for reporting results. 

The results of these calibration studies showed (1) how minor changes in the 
wording of observed behaviors listed on grade sheets can have a significant effect on 
ratings, (2) that large individual differences among raters exist, (3) that higher referent- 
rater than inter-rater reliability was achieved and (4) a significant increase in reliability 
was possible when improvements to the calibration sessions were implemented (wording 
of the grade sheet, quality of the video, better briefing of the event sets, etc.). The 
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frequency of the higher referent-rater reliability is important to note because there are 
differences among carriers in their views of whether a referent should be established and 
instructors trained to the referent, or whether achievement of their instructor’s inter- 
rater reliability was sufficient (George Mason University, 1996). 

A PC-based system that effectively presents a complete calibration session to 
individual evaluators was also developed. The system (1) provided the evaluator with 
a pre-briefing of the LOE that was to be presented, (2) displayed the grade sheet that was 
to be used, (3) presented an audio/video of the first event set, (4) collected ratings to the 
event set and the observed behaviors, (5) continued to present and collect ratings for the 
entire LOFT or LOE, (6) analyzed the evaluators’ ratings and presented each evaluator’s 
performance relative to other evaluators and to a referent score based on the fleet’s 
qualification standards and (7) allowed the evaluator to select any segment of the video 
and receive instructions as to what the qualification standards were and why a specific 
rating was deemed appropriate. This project produced a series of documents describing 
what it means to have quality data, why quality data are necessary for training under 
AQP and the methods for achieving quality in performance assessment (Johnson and 
Goldsmith, 1999a, 1999b, 1999c). 


14.5. Leadership/Followership 


By the mid-1990s industry data were showing that the CRM element leadership was 
weak within some flight crews, possibly compromising safe operations. So the question 
became how best to train this skill. In some academic areas leadership has been viewed as 
a linear, one-way (i.e. downward) action with the primary aim of task accomplishment. 
Another approach views leadership as an activity that involves both leaders and followers 
as they interact to accomplish goals. Followership skills, in this view, are as important as 
leadership skills to the safe and efficient performance of flight crews. Thus, the challenge 
was to train both leadership and followership to improve flight crew performance. In 
order to design an effective training program, background work required the 
development of a model of cockpit leadership to identify the critical skills necessary for 
safe operations. Then training would then be targeted to address these skills. 

This project analyzed cockpit performance to determine the behavioral components 
of leadership/followership skills that might be incorporated into training programs. 
From this analysis a model of cockpit leadership was developed. This model showed that 
effective and efficient cockpit operations requires crews to use the skills of envisioning, 
modeling, receptiveness, influence, adaptability and initiative.. The analysis also showed 
that these skills are common to both leadership and followership. This cockpit analysis 
suggested that a minimum amount of leadership/followership is required to achieve 
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a safe flight, that individuals constantly oscillate between leader and follower roles, and 
that weak leader or follower skills can be compensated for by other crew members 
(Mangold & Neumeister, 2001). 

Subsequent data from a line audit at a major carrier revealed that the components 
described in this model were related to overall crew effectiveness. The line audit data 
showed that, generally, in more severely abnormal situations, the crew tends to be 
less effective in exhibiting leadership/followership skills. Captains tend to be less 
likely to articulate a vision for the flight, meet company standards, obtain 
commitment from other crewmembers or be adaptable. Both captains and first 
officers tend to be less likely to initiate actions in response to an operational 
deficiency when the severity of the abnormal situations was high. Yet when handling 
a complex situation, but not an abnormal one, these crews displayed “outstanding” 
leadership/followership skills. When captains exhibited good conduct and high 
standards, first officers exhibited similar behavior. When captains were receptive, first 
officers were likewise receptive. 

Maintaining vigilance during the flight seemed to be dependent on the leadership of 
each crewmember at different phases. During pre-departure, takeoff, climb and cruise, 
vigilance was related with the captain’s envisioning, modeling and receptiveness. 
However, during the descent and approach phases, vigilance was related to the first 
officer’s conduct and standards. 

Workload and task distribution was dependent on the captain’s conduct, standards 
and receptiveness. Establishing guidelines for automated systems for all phases of flight 
was related to the captain’s articulating a vision, including the captain’s conduct and 
standards (modeling), receptiveness and use of interpersonal skills to obtain commitment 
from others. 

Based on the skill analysis model and this line data, requirements for a leadership/ 
followership training curriculum were identified. This included the development of 
classroom exercises extending beyond role-playing, the creation of event sets that 
addressed critical leadership/followership skills, and addressing company philosophy and 
policy issues. A training package was developed, which includes an instructor manual, 
student manual and training videos (Dunlap & Mangold, 1998a, 1998b). 


14.6. Advanced Crew Resource Management 


While high-level categories of CRM behavior such as Communications are often 
viewed by the inspector workforce as too subjective to measure, specific applications 
of these concepts, such as Conducts Preflight Briefing, often do not instill the same 
level of concern. Playing off this observation, the next study examined ways to 
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proceduralize CRM and integrate those new procedures into existing crew 
documentation. Working with a regional carrier, this project focused on the design 
and implementation of a prototype CRM training program based on a task analysis 
methodology. The study integrated CRM performance requirements into the standard 
operating procedures of the air carrier (Shultz, Seamster and Edens, 1997, Seamster 
et al., 1998). 

Procedural CRM means the implementation of specific calls, checks and/or 
guidance into one or more of the following: normal checklists, Quick Reference 
Handbook, Abnormal/Emergency Procedures, Flight Standards Manual, and additional 
Job Aids. This can be viewed as translating critical CRM principles into CRM 
procedures. In developing the Advanced CRM (ACRM) course data from line 
operations, instructor comments and ratings, and findings from the NTSB commuter 
safety study were used as the basis for the CRM training design. These data were 
incorporated into a proceduralized management system specifically tailored to the needs 
of the carrier. This ACRM course was given to pilots in the research fleet. Their 
performance in a LOE was assessed prior to this training and assessed after the training 
course was completed, with an appropriate time lapse. The central focus of this research 
was to determine if a proceduralized system would increase pilot performance, thereby 
hopefully increasing safety. This research proceduralized the following CRM skills in the 
operational environment of the carrier: Team Management, Crew Communication, 
Decision-making and Situation Awareness. 

The Quick Reference Handbook was rewritten to reflect proceduralized CRM. 
The normal procedures were rewritten to incorporate the proceduralized CRM. One 
fleet received this prototype ACRM while another fleet received the traditional 
CRM training that the airline currently offered. After training, the performance of 
both fleets was assessed in the simulator and on the line. The final evaluation of the 
ACRM training employed several data sources: Instructor/Evaluator (I/E) data from 
simulator and line, performance, subjective evaluation of ACRM by pilots in both 
fleets and direct jumpseat observations of crew performance in both fleets. Data from 
these sources revealed that the combination of specific CRM procedures that were 
both trained and incorporated into fleet SOP were effective in producing specific 
changes in crew performance. Combined data showed that the performance of the 
ACRM-trained fleet was consistently and significantly better than the traditional 
CRM.-trained fleet both in the simulator and on the line, and in all phases of flight. 
This also demonstrated the positive effects of ACRM transfer from the training to the 
operational environment. This project produced an ACRM manual to aid the 
development of other ACRM courses for specific operational environments (Seamster 
et al., 1998). 
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14.7. Facilitated Debriefing Techniques 


Debriefing after a LOFT session has been widely accepted as a valuable learning tool. 
The debriefing is a window on the entire CRM process. The LOFT simulation is a very 
busy, intense and sometimes stressful experience. Thoughtful discussion after the 
experience is necessary in order for the crew to sort out and interpret what happened 
and to consolidate the lessons learned into long-term memory in a form that can be used 
later in actual line operations. LOFT debriefings can demonstrate how well crews are 
able to analyze their performance along CRM dimensions. It is assumed that in order to 
implement CRM effectively in day-to-day line operations, crews must have the skill and 
the habit of analyzing their own performance in terms of NASA and the FAA wanted to. 
The FAA wanted to provide some guidance on LOFT debriefs and undertook work to 
produce such a document. The purpose of this project was to determine which 
techniques are actually being used by LOFT instructors, how effective the techniques 
seem to be, the extent to which those techniques are consistent with FAA Advisory 
Circular guidelines, how practical the guidelines are for real-world training and what 
obstacles instructors encounter in trying to teach according to these guidelines. Basically, 
this was a collection of best practices for conducting debriefs. Several major carriers were 
involved in this project. The debrief guide is available (McDonnell et al., 1997) 


14.8. The Generations of the CRM Advisory 
Circular 


Using Helmreich et als (1999) framing of six generations of CRM, the FAA’s viewpoint 
of the evolving conceptions of CRM, as reflected in updates to the advisory circulars and 
discussions among HQ FAA personnel, was, from the bottom-up viewpoint, as follows. 
The first generation, with its focus on crew personalities and management grids, did not 
provide a concept of CRM that the FAA was prepared to endorse. The FAA was not 
willing to regulate a system that relied on matching up personality traits, and waited for 
something more concrete to emerge. It was not until the development of the behavioral 
markers based on work done at the University of Texas at Austin by Dr Helmreich’s 
laboratory that the FAA felt CRM was defined to a level of detail that warranted more 
direct advisory guidance. Those markers were introduced early in the development 
of the Advisory Circular and have remained in all successive versions of AC 120-35 to 
this day. 

Although this chapter has focused on the CRM research funded in support the 
development of the AQP training program, it is the AQP program itself that has 
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provided the opportunities for these research-based tools and techniques to be 
implemented and evaluated in a timely manner. The regulatory flexibility, the enhanced 
data collection strategies, the annual meetings of all the AQP carriers, the 
encouragement of innovation over standardization, and the sharing of methods within 
the AQP carrier community have all facilitated the rapid evolution of CRM through its’ 
first six generations. The power of the FAA’s decision to support both voluntary and 
regulatory programs allows innovation to flourish. The prescriptive, regulatory 
programs establish minimal standards for all to meet, while the voluntary programs allow 
each air carrier to exceed those standards in a manner that reflects that carrier’s own 
unique safety culture. 
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Introduction 


To integrate CRM in Air Canada required organizational change which would ulti- 
mately permeate the culture of the airline. Organizational culture and change are 
weighty and controversial topics. For example, Google lists no fewer than 3,520,000 
results for changing organizational culture case studies. The complexities of organiza- 
tional culture and change are well documented (see, for example, Lewis, 1990; 
O’Donovan, 2006; Scheer, 2005). For this reason, the following account will focus on 
an anecdotal account of Air Canada’s integration of CRM, from the inside perspective 
of a participant and change-agent. 

The evolution of an airline’s culture will only occur when its pilots accept new values 
and practices. Changing culture requires individuals to realign their personal choices to 
those the organization has identified as preferable. The airline establishes the direction of 
this process but has little control over the pace of change. The ultimate criterion for 
change is when it reaches line practice. 

At Air Canada, our belief was that it would take eight to ten years to have the new 
values accepted by line pilots. Because these values included how one measures pilot 
competence and airmanship, the speed of innovation cannot be predicted accurately. 
The new set of beliefs cannot result from a company order, or be forced on the pilot 
group in any other quick fix. Rather, it depends upon a group’s collective realization that 
prevailing values no longer serve it adequately. 

For Air Canada, this change of culture evolved to the point that CRM was 
accepted not only as a feature of operations. It was pervasive, with values of CRM 
extending from the boardroom to the training and checking departments, into the 
safety department and ultimately onto the line. The development has proceeded to the 
point that the CRM meets the 4 Ps Weiner stipulates as indicators of application to 
line flying (Degani and Weiner, 1994, p. 44).These indicators include: 


m Philosophy: The airline management determines an overarching view of how they 
will conduct the business of the airline, including flight operations. A company’s 
philosophy is largely influenced by the individual philosophies of the top decision- 
makers, but also by the company’s culture. 


m Policy: Policies are broad specifications of the manner in which management 
expects operations to be performed. 


m Procedures: Procedures, then, should be designed to be as consistent as possible 
with the policies (which are consistent with the philosophy). 


m Practices: A practice is the activity actually conducted on the flight deck. 
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The two key stakeholders in airline CRM are the pilots and management. Without 
the line pilot’s support CRM cannot be integrated into the company culture. Man- 
agement’s contribution is to ensure that CRM is an integral part of all stages of the pilots’ 
professional involvement in the airline. This will begin at interview and extend to 
ground school, flight training, checking, actual line operations, captain upgrades and 
safety management. These are demonstrations of management’s active commitment to 
the CRM culture. Further, management will need to seek systematic feedback from the 
line pilots to confirm that the CRM philosophy, policy, procedures and practices are 
consistent, being supported by all departments and on the line. This chapter will outline 
how Air Canada’s pilots and management did so. 


15.1. Why CRM? 


In the late 1980s, it became apparent that the world’s airliners were involved in 
incidents and accidents that were not related to aircraft handling. The causal variables, 
as in the United/Portland and KLM/PanAm Tenerife accidents, were human. These 
human failures involved communication breakdown, a lack of teamwork and decision- 
making errors. For example, someone in the cockpit or cabin may have had critical 
information to avoid the accident but was unable to convey this to the captain in 

a timely manner. Proficiency in aircraft handling was no longer the solution to airline 
safety. 

Air Canada’s culture, like that of most airlines at that time around the world, revolved 
around a captain’s command and authority. It was a captain-centric model. The first 
officer (F/O) and second officer (S/O) were in subordinate and supporting roles. 
They were the captain’s followers, implementing directions but excluded from the 
decision-making process. 

No training course existed to prepare potential captains and in practice no emphasis 
was placed on the effective use of available resources. An F/O was expected to model 
appropriate values and practices derived from observation of and experience with 
captains on the flight deck. From this range of behaviors (the good, the bad and the 
ugly), the F/O was expected to create personal best practices. 

Check-pilots assessing F/O upgrades did not have a checklist of human factor 
behaviors to use as a tool to measure the candidate with any degree of objectivity. 
Rather, they were expected to evaluate the candidate on the basis of a universal personal 
judgment: “Would I put my wife and kids on an airplane with this person as the 
captain?” 

The simulator was used mainly as a realistic means of evaluating technical 
performance such as V1 cuts, rejected takeoffs (RTOs) and single engine approaches. 
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The emphasis in the simulator was on aircraft handling. Once three or four check-pilots 
had a look at candidates in both the simulator and on the line, the conditions for 
promotion had usually been met. 

What occurred at Air Canada followed what has been recognized as a seminal 
process over four stages. First, was the awareness phase, in which some pilots attended 
seminars and conferences, and heard from experts and other pilots about the need for 
human factors to be incorporated into airline training. These pilots then passed on the 
information within their company. A three-day ground school for all pilots was 
developed by an outside consultant in consultation with Air Canada line pilots to 
develop the knowledge, skills and attitudes about the value of CRM training. 

The second phase was one of practice and feedback: the design and introduction of 
a simulator-based training course, line-oriented flight training (CRM/LOFT) which 
incorporated the reaction of participants and presenters. From industry interaction and 
communication, such CRM/LOFT scenarios were compared and refined. There was 
recognition that such moves were a necessary and positive development. This led to the 
third phase, one of reinforcement by company, regulator and pilots. Before line pilots 
operate ina CRM manner as part of their culture and peer pressure to make it the norm, 
the company must support it by their philosophy, policy procedures and practices. 
Integrating CRM into an airline is an evolutionary process that takes eight to ten 
years before it seems to disappear as an individual item and becomes a way of doing 
business. This is the integration of CRM into the airline’s culture, the final phase (Dowd, 
1995, p. 1). 

CRM is not a recent phenomenon. Its principles have been practiced by the most 
effective crews since the beginning of aviation. Over the past three decades, refinement 
and standardization occurred. Instead of it being the practice of a minority of true 
leaders in the cockpit, it became part of pilot training. 


m In the late 1980s CRM started in the cockpit, as Cockpit Resource Manage- 
ment. Then it became Crew Resource Management, and, finally, Corporate 
Resource Management. Flight dispatchers, mechanics and managers embraced 
CRM as effective and efficient error management through good communica- 
tion, decision-making, feedback and conflict resolution, workload management 
and teamwork. 


m The other evolution since the 1980s is in a more objective evaluation of CRM. 
It has been quantified and measured like the rating of a pilot’s technical skills. 
Sets of human factor markers clearly identify the best practices, skills and 
behaviors that produce the safest aircraft operation. The Integrated Safety 
Management System (SMS) is the new vehicle to track, quantify and measure the 
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level of safety in an airline. CRM sits inside this system as mandatory behaviour 
(Blake, 2008—09, p. 3). 


m Today, pilots no longer question the assessment of their error management and 
human factor skills. They are prepared to recognize error chains, discussing 
events with reference to “the Swiss Cheese Model” (Reason, 1990) and poor 
CRM. 


m Just as pilots follow standard operating procedures (SOPs) because they recognize 
the procedures promote safety by minimizing error, they now accept the rationale 
of CRM. Pilots accept that 50% of a Line Oriented Evaluation is human factors 
based. This is a testimony to how far CRM has come in the past decade. 


15.2. First Steps: CRM/LOFT on the A320 


Air Canada’s first step towards a CRM-oriented training program began with CRM/ 
LOFT (Line Oriented Flight Training) on the A320. Its advanced glass technology 
presented a new perspective on flight management for Air Canada. Because the A320 is 
a relatively junior aircraft in terms of pilot seniority, it was regarded as an appropriate 
catalyst for a new way of doing business. The new focus was quite different from the 
traditional pilot proficiency check (PPC) where only handling proficiency was tested. 
This was a major shift in the philosophy that piloting skills or proficiency in maneuvers 
were sufficient to maintain safety. 

Statistics pointed to human factors being the cause of up to 80% of all accidents, 
prompting the shift from the classroom to the simulator for CRM. Skills taught in 
briefings were practiced in flight scenarios. The simulator became a CRM training tool. 

Along with this shift came the change of training. Instruction on means of coping 
with and preventing human error had to be reconciled with training to proficiency. 
Next, confirmation was needed that the training had been done up to company and 
Ministry of Transport standards. Originally, the first day of a pilot’s recurrent training 
included four hours in the simulator. The training department signed off the pilot as 
being proficient and the following day the pilot returned for another four hours in the 
simulator. This permitted testing by the checking department to confirm that the 
procedures could be demonstrated to an acceptable standard. 

The changes were that training to proficiency took place on the first day of simulator. 
Then, on day two, a CRM/LOFT scenario was flown. It focused on communication, 
decision-making, workload management, feedback and conflict resolution. The 
innovatory aspect of this new training is that it combined CRM with technical 
proficiency. 
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15.3. Focus of CRM/LOFT 


Another impetus in Air Canada came from professional curiosity. Pilots who were 
scheduled for the simulator early in the seasonal cycle posted the scenarios on the web. 
This had the positive effect of prompting line pilots to discuss what they thought was the 
best solution to the problem posed by the scenario. 

Here was a novel element for air crew. SOPs were still there to be followed, but now 
an evaluation and analysis of options was associated. A new pilot training had evolved at 
Air Canada. 


15.4. Integrating CRM Theory and Technical 
Evaluation CRM/LOFT 


When CRM courses were introduced to many airlines worldwide, they had an image 
problem. This was a negative influence. Pilot descriptions included “charm school” or 
“ivory tower intellectual gab.” Such views had to be countered, so the CRM/LOFT 
program at Air Canada was developed to combat these perceptions. 

Air Canada recognized that many line pilots resisted the change in curriculum and 
wanted to do V1 cuts and single engine approaches. They wanted to pass a pilot 
proficiency check (PPC) as they had always done, even though statistics showed that 
hands and feet flying was not the major cause of accidents. They wanted the security of 
a familiar and relatively comfortable routine. They knew what to expect and how to 
handle the technical event. Human factors and LOFT were the unknown. Concern 
developed within the pilot group over what to expect during a Line Oriented Flight 
Training (LOFT) simulator session, 

A solution was to provide a demonstrable link between theory and practice: between 
CRM and line flying. The simulator was used to practice and get feedback on the CRM 
skills taught in the initial CRM classroom setting. The simulator activity had a focus on 
learning, not checking. This alleviated some concerns. 

However, the resistance to change in the organization persisted. Pilots disliked 
moving away from V1 cuts, RTOs and single engine approaches. They knew what the 
standard was in the old system and knew they could attain it. So why change to a system 
with new expectations and standards of performance. Why put your license in jeopardy? 

It was not enough to be told that CRM/LOFT was non-jeopardy. There was 
a natural reluctance to accept the new training program. This is exemplified by initial 
reactions to group discussions. Having watched a video segment of their handling of an 
event, crews were quite reluctant to discuss what they might have done differently if 
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presented with a similar situation on the line. The question “What have you learned 
today that you can take back to line flying?” was often met with silence. There was 
a reluctance to commit, based on a fear of personal incrimination. 

Chance and good fortune, however, intervened at Air Canada. Coincidentally, early 
in the CRM program a well-respected captain who wrote a union-backed paper on 
flying the A320 titled “Land Green” supported the CRM/LOFT program. After the 
captain/author’s training session, he wrote that it was the first time in 20 years that he 
had come out of the simulator having learned so much that he could apply to the line. 
This one individual gave credibility to the program, carrying it through the initial 
teething process. 


15.5. LOFT Scenarios 


A well-designed LOFT scenario is vital. 

The intention is not to make crews commit errors but to give them practice and 
familiarity in handling the detection, correction and management of errors, omissions 
and lapses. Ultimately, the scenario needs to provide the opportunity for the crew to 
mitigate or minimize the error. The crew has to be given the opportunity to apply 
human error skills taught in the CRM training program, along with their technical skills 
(British Airways, 1993, p. 1). 

Being in a controlled non-jeopardy environment where a crew can walk away 
from a crash, allows the learning experience to be integrated with the line 
experience. 

The use of real-life scenarios from airline practice, experienced by a fellow pilot, 
gave further credibility. The application to line flying was apparent to all. For example, 
one LOFT involved a last-minute change of runways on a clear day. The new runway 
was fewer track miles to touchdown than the original (an event that happens often on 
the line). When presented with the runway change the captain and F/O would make an 
assessment of the opportunity being presented and the risk involved. The pilot flying 
would choose whether to ask for a longer downwind, accept the shorter track distance 
or ask for a hold to program the new approach. Task-oriented pilots took on the 
challenge of going straight in on the new runway. Often they would ask the pilot not 
flying to go heads down to program the approach while the pilot flying initially flew the 
approach visually. It could be done by a proficient, well-coordinated crew. However, this 
option often led to a rushed, unstable approach or a last-minute go-around when it was 
seen that an undesired aircraft state was imminent. 

The debriefing the video playback of the event was most illuminating. In their self- 
debrief, the crew could see that the decision was often not discussed between the two 
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pilots. Usually, one pilot was not fully committed to the course of action being taken. 
The workload management was hectic and often bordered on chaotic. The facilitator 
only had to ask at what point the advisability of a go-around had entered each pilot’s 

mind. 

Invariably the idea of going around had occurred initially to the pilot not flying. 
However, he rarely raised concerns. The pilot flying was so task-saturated that he did not 
consider the risk being taken until much later in the approach. Often the result was an 
unstable approach followed by a long landing or a last minute go-around. 

Discussions about why the pilots not flying did not voice their concerns centered on 
professional courtesy. “It looked like you were committed to landing.” Decision- 
making, communication, workload management and feedback/conflict resolution all 
flowed from such scenarios. Crews usually stayed long past the allotted debriefing time, 
or adjourned to the local watering hole to continue the discussion of what they had 
learned from their training. 

That a professionally trained crew could get into such an unsafe condition on a clear 
day, with a perfectly functioning aircraft, was carried back to the line for reflection and 
discussion. The obvious relevance and application to everyday scenarios that line pilots 
face emphasized the value of line-oriented training. 

Facilitators were given a three-day training course (developed in-house), on the 
techniques to draw out the participants and get them to analyze their effectiveness, 
strengths and weaknesses, and how the CRM skill sets contributed to the outcome 
of the event (Bertram et al., 1994). In addition, Facilitating LOS Debriefings: A 
Training Manual (McDonnell et al., 1997) became an essential tool in this facilitator 
training. 

The C-A-L model incorporates the following three main concepts which were used 
as a framework for effective debriefings: 


m CRM (C) 
m Analysis and evaluation of the LOS performance (A) and 


m Application to Line Operations (L). 


15.6. Advanced Qualification Program (AQP) 


The next step came when the CRJ Regional Jet was purchased by Air Canada. The 
Advanced Qualification Program was initiated on the CRJ with the intention of putting it 
on all types over time. The A320, EMJ and B767 are now AQP. However, the A330 and 
B777 are yet to be introduced to AQP. 
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The decision was made because the CRJ, like the A320, was a new aircraft to the 
fleet. It would have younger crews, more receptive to change because of their shorter 
history of experience with other practices. To provide perspective on the AQP, it needs 
to be seen as part of the overall mission statement. 


15.7. Air Canada AQP Mission Statement 


The Air Canada AQP is founded on the principle that the content of training and 
checking activities must be directly driven by operational aspects or line-flying. Areas of 
concern were identified by: 


m Air Canada Flight Operations (via flight crew reports, FCRs or line checks) 
m Flight Safety (as a result of the British Airways Safety Information System (BASIS)) 
m Air Safety Reports (ASRs) or 


m Flight Operations Quality Assurance (FOQA) data analysis, part of the training 
program. 


The Air Canada mission statement policy clearly identifies that training will be line- 
generated, based on data collected from operations and practices. This ensures that 
procedures regarded as confusing, misleading or not properly understood can be 
rectified. 


15.8. Elements of Air Canada’S AQP 


The key CRM aspects of the AQP are: 


mw AQP will integrate the training and evaluation of CRM during all applicable 
phases of the syllabus. 


m Substandard performance on CRM factors will be corrected by additional 
training. 


m In AQP, demonstration of proficiency in maneuver-oriented technical skills 
such as V1 cuts will be a necessary, but insufficient, condition for pilot 
qualification. 


m To successfully complete a LOE, pilots must demonstrate proficiency in both tech- 
nical and CRM skills. (Air Canada AQP, 2005 pp. 1-2) 
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Fully 50% of the evaluation of a LOE rests on the pilot’s demonstration of CRM 
skills. The methodology for evaluation derives from the Airbus Industrie AQP 
Methodology (ACOPI/ADOPT) (Air Canada AQP, 2005, pp. 3.2). The measurement 
of human factor skills is now accepted as a valid, consistent, reproducible evaluation. 

Considering that the initial CRM courses were regarded initially by some line 
pilots as charm school, there is a true culture change with the current acceptance by 
line pilots that their license might not be validated based on unsatisfactory CRM 
skills. 


15.9. Annual Recurrent Training (ART) 


Largely as a result of the findings of the enquiry into the Dryden crash of Air Ontario, an 
Air Canada feeder (now Jazz), the annual recurrent training is done in conjunction with 
the flight attendants (Moshansky, 1992, p. 1236). 

These findings included determinations about safety awareness: 


m The Department of Transport would develop and implement a mandatory and 
comprehensive education program for all air crew engaged in commercial oper- 
ations, including an integrated program for cockpit crewmembers and cabin 
crewmembers. 


m All cabin crewmembers would be given sufficient training to enable them to 
recognize potentially unsafe situations both in the cabin and outside the aircraft. 


The evolving CRM philosophy was that by using all available resources, flights will 
be safer. Thus, a policy was put in place to train the flight crew to utilize the flight 
attendants as the ears and eyes behind the cockpit door. 

After 9/11 this became even more pronounced as the secure cockpit door 
procedures inhibited the pilots from freely accessing the cabin. In the Air Canada ART 
half a day is scheduled for a group session between the pilots and flight attendants. This 
exercise introduces flight attendants to CRM models. It also provides an opportunity to 
apply CRM in the analysis of past accidents and in developing team effectiveness and 
cohesiveness. 

Air Canada procedures call for pilots to actively inquire about what flight 
attendants can observe from the back of the aircraft regarding the engines and aircraft 
structure. 

An outline of the Air Canada Error Management System along with the CRM skills 
is presented in a pre-work package to the pilot and flight attendant attendees. The 
outline acts as a refresher and a methodology to examine the real-life scenarios included 
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in the package. This pre-course workbook must be completed by a participant before 
attending ART. Air Canada, 2007-2008 (ART). 


15.10. The Error Management System 


The “Swiss Cheese” Error Management System (Air Canada FCTM, 2008, p. 27) was 
designed as a tool to help visualize how crewmembers can manage threats and errors. 
One end of the model represents the flight, while the other end represents a potential 
incident/accident. In between, there are resources or lines of defense used to avoid, trap 
and mitigate the consequences of errors. 

The Error Management System uses the premise associated with James Reason’s 
“Swiss Cheese” model (Reason, 1990) based on the premise that accidents occur as 
a result of concurrent failures (Figure 15.1). The colored slices of the model represent 
the various resources that pilots have to manage a safe flight. The holes within the 
resources represent the system deficiencies and errors caused by human limitations, or 
intentional non-compliance with standard operating procedures. 

The Error Management System illustrates that there is a redundancy when dealing 
with threats and errors. If an error penetrates one slice or resource, it can be blocked or 
arrested by another. The probability of an error causing an incident or accident increases 
as the various resources are penetrated. 


Figure 15.1 The “Swiss Cheese” Error Management System (Reason/Volant) 
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15.11. CRM Skills 


The essential skills covered in the Air Canada program are (Dowd & Shrindruck 1999): 


Leadership 

Preparation and planning 
Briefings 

Monitor and feedback 
Situational awareness 
Communication 
Decision-making 
Workload management 


Crew performance 


An application of these skills to practice can be found in the ART 2008, the 
Helios Airways crash. The pilots misidentified the pressurization warning and lost 
consciousness while the flight attendants knew there was a decompression. The 
case study is used to exemplify the Error Management Model and to visualize why 
the errors on the flight were not caught. The analysis enables an analysis of errors 
in policies, procedures, flows, checklists, automation, CRM skills and aircraft 
handling. 

Positive reinforcement of good CRM skills and error management are shown 
through the Peach Air B737 flight between Ostend and Dover (also a pressurization 
problem). The conclusion is that “the successful outcome of this incident was not by 
chance.” The CRM skills performed by the crew, in particular the first officer, ensured 
the error (the pressurization problem due to the crack in the aft cargo door) was trapped 
and that the consequences of the error were mitigated. The first officer successfully 
applied a number of CRM skills: leadership, situation awareness, communication, 
decision-making, workload management and crew performance. At Air Canada the 
ART facilitator takes the combined pilot/flight attendant group through an analysis 
using the model as a guide. CRM skill sets are posted. 

Thus, two similar incidents are examined, one having a positive outcome due to the 
application of effective error management and CRM skills. The other demonstrates less 
than fully effective error management and CRM skills. 
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Similarly, CRM can be drawn from the professional handling of the Air France A340 
evacuation in Toronto, where the aircraft overran the runway and ended up in a ditch at 
the end of the runway. Actual problems are encountered when passengers retrieve their 
carry-on luggage in spite of warnings not to do so. Such actions are analyzed with the 
intent of how even a crew highly trained and motivated can face irrational behaviors or 
errors in an emergency. Forceful commands need to be employed to trap or mitigate 
these actions. 


15.12. Flight Data Analysis (FDA) 


Flight data analysis looks at the practice on the line. The two most often seen are unstable 
approaches and long landings. If the SOPs are not being adhered to or are not effective it 
then reverses the Four Ps model so that one backtracks. The analysis indicates why the 
practice exists then tries to develop a procedure or SOP which would lead to the ideal 
policy. Ultimately, this generates a philosophy of safety. 

This proactive tool, FDA, gives an accurate view of the day-to-day margin of safety 
by monitoring the trends occurring and using a formula of risk assessment to project 
future areas that need addressing. However, the data do not indicate what to do to create 
a safer operation. Unfortunately, data alone provide a picture of reality but not how to 
create a safer system. 

Administered by ACPA (association/union) gatekeepers, all elements of the flight are 
de-identified. The information gained is used as a learning tool which is given to the 
company as an instant feedback on daily operations. Over 100 parameters are monitored 
on the A320 fleet. Limitations such as long landings are automatically highlighted by the 
computer program for investigation. 

However, the FDA system does not use human factor questionnaires or interviews 
with the operating crew to identify why the crew felt a certain series of events took 
place. The only reason a gatekeeper would call the crew is to clarify what happened, not 
what they felt were the human factor or technical root cause of the event. While this 
helps the safety reporting maintain its non-threatening parameters, it leaves crew 
feedback out of the equation. It is a valuable resource in tracking trends and from the 
data, gatekeepers along with management can propose corrective actions within the 
context of the Safety Management System. 

One trend was an increase in long landings, one flight landing more than half way 
down the runway. The data revealed the clear relationship between stable approaches 
and long landings. SOPs stipulate that an approach must be abandoned if it is not stable 
by 500 ft in visual conditions and at 1,000 ft on an IFR approach. The data revealed that 
this was not always being followed. Further, many of these long landings were happening 
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at busy airports with long runways. Additionally, ATC were requesting 180 knots until 
crossing the outer marker. If there happened to be a tail wind or the flight had chosen 
a low drag flap setting (a decelerated approach as recommended by SOPs to save fuel) 
then the occurrence of fast long landings increased. 

A review of the limits for a stable approach with flight crews did not yield a lack of 
knowledge, skills or attitudes. Simulator scenarios which were designed to create a fast 
long landing were not effective in training crews to abort landings due to unstable 
conditions (often because it was the simulator and not real life). It was thus a challenge 
with no jeopardy. It might lead to crews trying the same thing on the line and thus be 
negative learning. 

In a LOFT scenario a trigger event starts the action for a series of human factor 
events. For example, ATC could keep the flight high and fast then turn them in close to 
the marker or ask the crew to change to another runway with fewer track miles to go 
before landing. 

In the class room, it is relatively easy to develop a statement stating your concerns. 
“Tam uncomfortable with making this approach under the current weather conditions. 
Should we hold for the CB to pass the field or divert to our alternate when the fuel 
reaches xx kilograms?” 

This unsolicited type of statement is less than ideal. Rather, the captain should be 
constantly inquiring about and exploring the options with his F/O. This leads to better 
teamwork and a continual reassessment by the crew. Conflict between the two pilots is 
avoided. Often during a debriefing from a simulator scenario, F/Os were asked if they 
felt that they were part of the decision-making process. After a negative response, they 
were then asked if they were comfortable with the path taken. The second negative 
response usually surprised the captain. 

A major change in procedures came from these observations. The development of an 
SOP verbal “stable or unstable” call at 500 ft was tested on the A320 fleet for six months 
in an attempt to decrease unstable approaches that resulted in long landings. The SOP 
mandates a go-around if at 500 ft the call is “unstable.” The FDA program showed an 
increase in go-arounds due to unstable approaches. The “stable/unstable”’ call was 
expanded to all fleets in the company. Looking at practices led to modifying the standard 
operating procedure, resulting in a greater margin of safety. 


15.13. The 80% Factor? 


We know that in 80% of all accidents the captain is the pilot flying (NTSB report 1994 
SS-94-01). New policies and procedures are needed to eliminate this imbalance. Why 
would this percentage be so skewed? 
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It may be because most flights are turn-arounds and the captain may be choosing to 
do the leg with the marginal weather. Another reason may be that the captain may be 
flying the first leg to the unfamiliar airport, leaving the return leg to the home base 
airport to the F/O. However, this still does not account for the great discrepancy in 
accident rates by the pilot flying. Perhaps the F/O needs a tool such as SOPs as indicated 
above, rather than using assertiveness as a trigger to change the course of a flight. 


15.14. Line Operations Safety Audit (LOSA) 


A Line Oriented Safety Audit (LOSA) is conducted every three to four years. The 
purpose of collecting information on the day-to-day compliance of the pilot group to 
SOPs is essential to give feedback on what procedures are working and being accepted 
by line pilots and those that need further refinement. Volunteer pilot observers, along 
with observers from the LOSA Collaborative Company, are trained to collect human 
factor and technical data from line flights. The analysis of the data helps to point to line 
operations where the safety barriers are strongest and the defensive barriers are weakest. 
The management or mismanagement of risk posed by threats and errors is rated for 
frequency where the aircraft enters an undesired aircraft state (UAS). The fleets are 
compared both within Air Canada and with the 8,000 flights from other carriers 
contained in the LOSA database. 

The role of the standards and training departments is to determine where action is 
required and the form of action to be taken. Such activities could be an awareness 
campaign, changes in procedures or incorporating different material in the training 
syllabus. 

Surprisingly, there is an unexpectedly high rate of intentional non-compliance. 

For example, if a crew does not make the altitude calls at 10, 20 and 30 thousand feet, 
and at 1,000 feet before level-off, this is considered intentional non-compliance. 
Another example would be if the crew discussed a deviation from established procedure. 
Of 8,000 flights in the LOSA archive, those crews with an intentional non-compliance 
error are three times more likely to commit other types of errors and mismanage errors 
(Guillemette, 2008—09). 

Knowing that a segment of the pilot population is not adhering to the SOPs is 
valuable information. It highlights that some procedures need revision or further 
explanation as to the philosophy behind the SOP. 

Both FDA and LOSA are valuable safety tools in seeing inside the company’s 
cockpits. If the company philosophy and procedures are not being implemented by the 
line pilot, the safety system needs to examine why the Four Ps outlined by Degani and 
Weiner are not functioning effectively (Degani and Weiner, 1994, p. 63). 
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15.15. CRM Mission Statement 


Air Canada’s New Crew Resource Management Program is designed to enhance the 
safe operation of our flights. Systemic barriers have been erected to limit the risk of an 
accident, but no system is perfect. As pilots, we are the last line of defense and it is vital 
that we learn from the experience of others. CRM at Air Canada is designed so that 
participants will be able to identify and take action to “plug the holes” in Air Canada’s 
flight safety defenses. CRM teaches proactive accident prevention strategies that can be 
applied to every flight. 

Error management is the core around which the course components are tied. 
Human error is inevitable. However, with proper training we can reduce its frequency, 
correcting the error before it has consequences, or limiting its impact when it happens. 

Air Canada’s comprehensive new CRM program includes the following five 
elements: 


m The Introductory CRM course (primarily for new pilots at Air Canada) 
m The Introduction to Command CRM module 

m LOFT training 

m The Annual Recurrent Training CRM module and 

m AQP program training. 


CRM is part of all training events and evaluations and is embedded in Air Canada 
Culture (Auerman, 2001). 


15.16. Hiring Board Practices 


Human factors was integrated into the structure and operations of the hiring board. 
The philosophy was that the best selection and training was conducted by the military. 
Three senior management captains did the interviews of the new hires. So, the 
primary source of candidates was the Canadian military. 

As this source became limited when fewer military pilots were being trained, 
two-year college diploma courses were used as a source of recruitment. These programs 
were developed to train potential candidates. Some pilots were hired directly from 
the college with 250 hours to become second officers. The philosophy was that 
employing pilots before they had developed any bad habits, they could be trained 
into the Air Canada model pilot. This was seen as the safest route. 
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However, between the period of having second officers and before the company had 
cruise pilots came a time when pilots were hired directly into the right seat of Air 
Canada’s jets. This resulted in a rethinking of the experience level of the new hires. 
Thus, the philosophy of molding inexperienced pilots into the Air Canada image and 
values changed. The airline hired more experienced pilots (with more than 2,500 hours’ 
flying experience). Included in this group were the Dash 8 pilots from Air Canada’s 
feeders. Policy also required that a more diverse group of candidates be hired, 
representing Canada’s multi-ethnic community. 

Along with this philosophy change evolved a policy shift in which line pilots of all 
rank were on a 20-person hiring pool. In essence, pilots were hired by the peers with 
whom they would be flying. Three pilots were chosen to do each interview. The 
questions targeted actual experiences of the candidate, not hypothetical scenarios. A 
question such as “What would you do if you saw your captain drinking within the legal 
limit before a flight?” became “Tell us about an in-flight emergency or incident and how 
you handled it.” 

The direction of the interview was to probe the candidate’s values concerning 
human factors, specifically teamwork, communication, standard operating procedures 
and customer service. The use of a standardized format helped keep the hiring process 
equitable and systematic. However, after about one year of using this same format it 
became apparent that many of the feeder pilots knew the questions they were to be 
asked. The selection interview became less effective. Pilots coming off the street did not 
know the questions and were at a disadvantage. 

The procedure of psychological testing has been used with varying degrees 
of success. The tests ranged from an interview with the company doctor to the 
Myers-Briggs and a psychological test which measures compatibility to the job and 
traits matching the job description. In Air Canada the human factors of hiring 
has become more systematic and less subjective. Technical proficiency is no 
longer the sole criteria for employment but rather a combination of CRM and 
airmanship. 


15.17. Introduction to Command (ITC): The 
CRM Policy of Air Canada Toward Captaincy 


The policy of a company towards its pilot in command is the legally mandated and 
ethically required responsibility, authority and accountability of its captains to perform 
their duties in a safe and efficient manner. The inclusion of CRM in the statement of the 
captain’s authority makes CRM skills pivotal to the way it does business. 
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15.17.1. Captain’s Authority 


Authority is the power granted to direct, perform and supervise assigned tasks and 
responsibilities. The captain’s authority refers to the rights and responsibilities of the 
pilot in command as set out in the Canadian Aviation Regulations (CARs) and the Air 
Canada Flight Operations Manual (FOM). Co-authority for the dispatch of a flight 
exists between the captain and the flight dispatcher. Air Canada captains are required to 
apply Crew Resource Management skills in dealing with other crewmembers and 
departments to ensure cooperation, understanding and action in matters relevant to the 
operation of the flight. The final decisions and actions taken for the safe operation of the 
flight rest with the captain. In making such decisions, captains utilize the elements of 
command. These elements (Air Canada, 2008, p. 2) are: 


m Knowledge of regulatory, aircraft operating manuals and flight operations manuals 
m Skill in aircraft handling both on the ground and in the air and 
mw CRM. 


It is acknowledged that a captain will inevitably make errors. That is why the latest 
generation of CRM training deals with threat and error management. Being aware and 
using CRM skills will lead to a more effective use of a captain’s resources and result in 
a safer flight. The most important skills for captains are leadership and team building. 
They must manage their team and utilize the resources at their disposal. 


15.17.2. Contribution to Policy and Procedure 


A more sophisticated form of management is the captain’s critical assessment of the 
company’s operation. In the course of line operations, flight crews may be presented 
with the results of an ineffective policy or procedure. The Air Canada captain is 
expected to report circumstances where an existing procedure appeared ineffective or 
inefficient. Captains should provide constructive suggestions that will aid in rectifying 
problems or improving procedures. The feedback from flight crews 1s regarded as critical 
to improve Air Canada’s operation. 

Hawkeye Reports are used to provide instant feedback. These data link messages are 
coded to specify the different problems facing operational crews. Such problems may be 
non-SOP pushback procedure, a faulty de-icing procedure, etc. 

At weekly meetings, managers review the Hawkeye Reports to identify which 
procedures are not being implemented and those which need further refinement. 
Detailed flight crew reports are submitted via computer. Included are constructive 
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suggestions which offer insights into problems encountered by line crews in everyday 
operations. Often inquiries need to be made from departments outside flight operations. 
It is the time taken to get feedback to the line crews that is critical in making the system 
credible to pilots. 

The Introduction to Command Course gives captains the responsibility of 
identifying ineffective policy and confusing procedure. The airline is a complex 
organization, relying upon humans for its effectiveness and efficiency. The command 
course emphasizes that CRM is the captain’s most effective tool in accomplishing an 
improvement in the way we do things (Air Canada, 2008, ch. 5.1.1). 


15.18. Conclusion 


Effective CRM has been practiced by the most efficient crews since the beginning of 
aviation. CRM from the late 1980s had the unique quality of having originated in the 
cockpit. It was bottom-up, not top-down, change. 

First, it was seen as Cockpit Resource Management, then it became Crew 
Resource Management. Finally, it expanded in the organization to become Corporate 
Resource Management involving dispatchers, mechanics and managers. As the 
concept refined, it expanded in scope. 

CRM encompasses effective and efficient error management through 


m good communication 

m decision-making 

m feedback and conflict resolution 
m workload management and 

m crew performance. 


Over the last ten years CRM has been quantified. It is now measured like a pilot’s 
technical skills. Sets of human factor markers clearly identify the best practices and 
behaviors or skills sets that produce the safest aircraft. Today, pilots no longer question 
the rating of error management and human factor skills. The fact that pilots accept that 
50% of a Line Oriented Evaluation is human factors based is a testimony to how far 
CRM has come in the past decade. 

This acceptance can only occur if the airline supports CRM by its philosophy, policy, 
procedures and practices. Integrating CRM into an airline’s culture is an evolutionary 
process that must be monitored and integrated into all facets of the airline’s operations. 
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The Integrated Safety Management System (iSMS) is the new vehicle to track, 
quantify and measure the level of safety in an airline. It is a shift from reactive to proactive 
safety management; a systemic, explicit and documented process for the management of 
safety risks and an effective means for involving all employees. CRM sits inside this 
system as mandatory behavior (ART iSMS, 2008—09, p. 52). 
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The sole purpose of the investigation of an accident or incident investigation shall be the 
prevention of accidents and incidents. It is not the purpose of this activity to apportion blame 
or liability. 

ICAO Annex 13, Paragraph 3.1 (ICAO, 2001) 


Introduction and Background Information 


As with accident investigation, the ultimate objective of Crew Resource Management 
(CRM) is to prevent accidents. According to the US Federal Aviation Administration 
(FAA) Advisory Circular (AC) 120-51, Crew Resource Management Training, first 
published in 1989, CRM contributes to accident prevention through improved crew 
performance, including team management, the use of all available resources, and 
through addressing the challenges of human computer interaction and workload (FAA, 
2004). 


Background 


Accident investigation has played a significant role in the development and adoption 
of CRM in the aviation industry. The concept of CRM gained momentum 
following the 1978 United Airlines’ fuel starvation accident at Portland, Oregon 
(NTSB, 1979b). Through this investigation, the National Transportation Safety 
Board (NTSB) issued its first recommendation on CRM, resulting in an initial FAA 
operations bulletin.“ Also in 1979, the National Aeronautics and Space 
Administration (NASA) sponsored a workshop devoted to methods to reduce 
aviation accidents, with a focus on CRM (Cooper et al., 1980). Subsequent 
accidents in the 1980s and 1990s prompted a series of further NTSB 
recommendations and FAA actions on CRM, as operators began to adopt the 
concept. However, it wasn’t until 1997 that FAA regulations actually required” CRM 





* In response to NTSB Safety recommendation A-79-047, in November 1979 the FAA 
issued an operations bulletin to all air carrier operations inspectors to urge operators to 
indoctrinate flight crews on flight deck resource management. 

> CRM was a component of the FAA knowledge requirements for airman certification 
in earlier years. See Title 14 of the US Code of Federal Regulation (CFR) Part 61, 
Section 61.155 and 61.65, and Part 121 Section C141.1. 
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training for scheduled air carriers (FAR® Part 121 operators), followed by fractional 
ownership operations (FAR Part 91 Sub-Part K) in 2003, and for FAR Part 121 
Advanced Qualifications Programs (AQP) in 2005.’ 

Since 1979, the industry has made significant progress in instituting CRM concepts 
as a core component of training, integrating the concept through both ground courses 
and simulator sessions (Line Oriented Flight Training; LOFT). This progress has had 
a decidedly positive effect on safety. The Sioux City, SD accident involving United 
Airlines flight 232 is only one example of effective CRM, in which the Safety Board 
lauded the crew’s performance (NTSB, 1990). “The Safety Board views the interaction 
of the pilots, including the check airman, during the emergency as indicative of the 
values of cockpit resource management...” The Board also stated that “under the 
circumstances, the [United] flightcrew performance was highly commendable and 
greatly exceeded reasonable expectations.” 

In a recent public hearing regarding the successful forced landing on the Hudson 
River, New Jersey, of US Airways flight 1549 on January 15, 2009," the captain testified 
to the significant role of CRM in the outcome of the flight. In this accident, the plane 
suffered loss of both engines shortly after departure from New York’s LaGuardia Airport, 
and despite landing in the Hudson River, there were no fatalities. In post-accident 
interviews, the captain credited the CRM training provided at US Airways that gave 
them the skills and tools they needed to build a team quickly and open lines of 
communication, share common goals and work together. 





° FAR is the abbreviation for Federal Aviation Regulation. Offcially, FARs are codified in 
the CFR as Title 14 CFR, and then the related section number. For example, FAR Part 
121 could also be referred to as 14 CFR 121. Both conventions will be used in this chapter. 
7 Part 121 flight crewmember since March 1998 (14 CFR Section 121.404); Part 121 
flight attendant or aircraft dispatcher since March 1999 (14 CFR Section 121.404); Part 
91 Sub-Part K since 2003 (14 CFR Section 91.1073); and Part 121 (under AQP) since 
November 2005 (14 CFR Section 121.907). Note: A Special Federal Aviation 
Regulation (SFAR No. 58) for AQP was first issued in 1990, which is a voluntary 
program for Part 121 operators. Since its inception AC 120-51 encourages CRM 
training for Part 135 operators, but there is still no regulatory requirement for Part 
135 operators to incorporate CRM. 

8 The accident was still under investigation at the time of writing this chapter. Reference 
NTSB public hearing regarding flight 1549 with Captain Sullenberger on June 9, 2009. 
Access interview with captain in the NTSB Group Chairman’ Factual Report: 
Operations/ Human Performance, May 15, 2009, Docket # SA-532, Exhibit #2-A, p. 30. 
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The Accident Investigator’s Perspective 


The value of documenting and learning from “what goes right” as well as “what goes 
wrong” is not contested (Reason, 2008). The NASA Aviation Safety Reporting System 
(ASRS) database includes numerous examples of the mitigating effects of CRM in 
situations that may easily have led to incidents or accidents without this type of 
intervention.” However, the accident investigator’s perspective is mostly one of 
hindsight, of trying to determine how to refine the system, knowing that there is always 
room for improvement when it comes to human lives. NTSB data indicate that, since 
1979, the Board has specifically cited lack of effective CRM in 20 accidents operated 
under FAR Part 121 and six accidents operated under FAR Part 135.'° These data 
warrant improvement in how flight crews approach and adopt the concepts of CRM, 
and in the system designed to reinforce associated behaviors and attitudes. 

This chapter will address two main issues from the perspective of the accident 
investigator, with the goal of increasing the effectiveness of CRM in reducing accidents: 
individual accountability within the context of team performance, and organizational 
accountability and indoctrination. Although the scope of this chapter is limited to the 
aviation industry, these principles apply equally to other industries, especially other 
modes of transportation and high-reliability organizations operating in high-risk 
environments. 

The first issue is individual accountability within the context of team performance. 
With the team perspective, crew performance profits from increased synergy and 
increased opportunity for error prevention. However, acting as a team does not 
relinquish crewmembers of their individual roles and responsibilities. In reviewing 
recent accidents, the effect of CRM on crew performance would benefit from 
broadening the focus to include other aspects integral to CRM that emphasize individual 
accountability within the team context. The first part of this chapter will review 
performance markers in FAA’s AC 120-51 (Version E) and will use recent accident 
examples to illustrate the value of greater emphasis of the aspects of leadership, adherence 
to standard operating procedures (SOPs), active monitoring and fatigue management. 

The second issue is organizational accountability and indoctrination, through 
promoting a systems approach to accident investigation, and holding the organization 





” See http://asrs.arc.nasa.gov 

'° These data highlight the need for charter operations to adopt CRM training. In 
2003, the NTSB issued Safety Recommendation A-03-52 for the FAA to require 
CRM for Part 135 operators, and in, 2006 added this recommendation to the 
NTSB Most Wanted List of Transportation Safety Improvements. 
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accountable for full alignment of the principles and values of CRM system-wide. 
Although CRM is carried out by the front-line employee, the effect of training is 
minimal without organizational policies and procedures to promote and support the 
crew’s ability to adopt and maintain CRM principles. This section of the chapter will 
focus on the systems approach to evaluating the role of CRM in accident investigation, 
and will provide examples of accidents that highlight the significant role of the 
organization in contributing to the support and reinforcement of CRM behaviors and 
attitudes on the front line. 

This chapter embraces the perspective that the objective of CRM is to reduce errors 
and increase safety (FAA, 2004). When carried out in a meaningful manner, the 
behaviors and attitudes of communication, teamwork, monitoring, etc., are simply 
a means to reduce errors and increase safety (Helmreich, Merritt and Wilhelm, 1999). 
Throughout this chapter, we will advocate that the success of CRM is dependent upon 
a focus on safety at both the individual and organizational level. With this in mind, we 
present you the perspective of the accident investigator. 


16.1. Individual Accountability within 
the Context of Team Performance 


Historically, in “promoting the use of all resources on the flight deck,” the focus of 
CRM was on the potential of the captain to assert inappropriate levels of authority 
without requesting or responding to communication from other crewmembers, and for 
other crewmembers to fail to assert themselves or communicate out of fear or respect for 
the captain in this hierarchical arrangement. Early accidents support this focus, such as 
the accident in Portland, OR mentioned earlier (NTSB, 1979b). In fact, the NTSB’s 
initial recommendation on CRM in 1979 (A-79-047) called for, “...particular emphasis 
on the merits of participative management for captains and assertiveness training for 
other cockpit crewmembers.” 

From the accident investigator’s pempecnve, although continual reinforcement of this 
historical concept is still important, | recent accidents point towards the need to focus also 
on other integral components of CRM. Leadership, adherence to standard operating 
procedures (SOPs), active monitoring and managing fatigue are all integral components 
of CRM equally valued for their contribution to crew performance (FAA, 2004). These 
components highlight the need for individual responsibility within the team context. 





11 See NTSB (2001), in which a new-hire first officer failed to assert himself 
appropriately with a senior management captain. 
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This section will review performance markers cited in AC 120-51, using accident 
examples to illustrate the value of a greater emphasis on these particular components. 


16.1.1. Leadership 


Being pilot-in-command of an aircraft is all about leadership, and embodies a host of 
behaviors and attitudes. Leadership is not simply having responsibility for the outcome 
of the flight, but for taking the authority to make the difficult and final decisions, and 
when the situation requires, making a command decision. According to 14 CFR 91.3, 
“The pilot in command of an aircraft is directly responsible for, and is the final authority 
as to, the operation of that aircraft,” and “In an in-flight emergency requiring immediate 
action, the pilot in command may deviate from any rule of this part to the extent 
required to meet that emergency.” As in the case of US Airways flight 1549, the captain 
also testified to the importance of command authority and its role in effective CRM. 
During the event sequence, upon loss of both engines and the crew’s inability to restart 
the engines, the captain made a command decision regarding the landing location. 

The reason to highlight the value of the role of the pilot-in-command as a strong 
leader is that some have erroneously believed or have misinterpreted CRM to mean 
“leadership by committee.” The role of captain must be assumed as a personal 
responsibility for each flight. The FAA voiced concern about captains ceding their 
leadership authority in a response letter to the NTSB regarding Safety 
Recommendation A-86-019: 


An over-riding concern of the FAA’s efforts in this program is to underscore the role and 
ultimate responsibility of the captain in a practical manner. The pilot-in-command of an 
aircraft has a very significant responsibility and must retain unchallenged authority for the 
conduct of a flight. Certainly the pilot-in-command must integrate the other flightcrew 
members in the cockpit procedures and decision-making processes. However, the teamwork 
concept must always stress the role of and authority of the pilot-in-command, and conversely, 
educate the pilot-in-command of the need to encourage and receive feedback from other 
flightcrew members... The consequences of an ill-conceived cockpit resource management 
program could very well be a captain who does not command properly or an overly assertive or 
dominant first officer who contributes to the problem and not the solution. 

(FAA, 1986) 


Recent accident investigations have revealed reluctance on the part of the captain to assert 
his/her authority. One example is the December 2005 accident involving Southwest 
Airlines flight 1248 (NTSB, 2007d). On this winter evening, a B-737-700 overran the 
runway at Chicago Midway Airport. During the en route phase of the flight, the captain 
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indicated his reluctance to land with the minimal runway stopping margins presented by 
the on-board performance computer (OPC), and asked the first officer to check the 
weather reports for the various potential divert locations. The first officer provided indirect 
pressure to continue with the landing, reiterating several times that the flight operations 
manual (FOM) authorized the landing based on the OPC calculations. The captain also 
stated discomfort in using the autobrakes for the first time for this landing, '” and later stated 
his decision to use the manual brakes. Instead of following through with his initial 
impressions regarding decisions affecting safety, he allowed the first officer to convince him 
otherwise: “Well, keep talkin’, I, I, I guess we could do it. Let’s, let’s see what the conditions 
are up there. We’ll do it. If you're comfortable with that, I am too.” 

AC 120-51 states that as a part of the captain’s pre-flight briefing, “bottom lines for 
safety are established and communicated.” In this case, the captain failed to establish with 
the first officer acceptable minimums for safety and failed to take a leadership position in 
ensuring safety—to use his authority to carry out the actions that seemed more sensible and 
safe. Instead, he only stated his level of discomfort, and allowed the first officer to talk him 
into a more risky option, as if the decision-making process required consensus. NTSB’s 
accident report cited the contributing role of the crew’s failure to divert to another airport. 








' The use of autobrakes was a new company procedure that had not previously been 
used by this crew, either on the line or in simulator training. 
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Research in the area of decision-making indicates a tendency for more risky 
decisions among groups than among individuals. When reaching consensus, the final 
decision polarizes towards the most risky attitudes among group members (Mosovici 
and Zavalloni, 1969), regardless of whether the leader or another group member holds 
the more risky attitude (Hoyt and Stoner, 1968). This research highlights the need for 
the captain to counteract this tendency by taking a strong leadership position as the final 
authority for setting the safety minimums for each flight and to use information from 
others to increase safety margins, versus decrease them. Again, the intention is not for 
group consensus on the flight deck, which tends towards a more risky option, but for the 
pilot-in-command to take authority in reaching the safest decision given all available 
information. Best-selling author John Maxwell says leadership boils down to one thing: 
“Leadership is about influence. Nothing more, nothing less” (Maxwell, 2002). 

Shuttle America flight 6448, operating as a Delta Connection, is another example of 
ceding leadership responsibility (NTSB, 2008). This accident, which also involved 
a runway overrun in winter conditions, occurred in February 2007 at Cleveland Hopkins 
International Airport, OH. Fortunately, there were only minor injuries. Heavy snow was 
falling at the airport which reduced visibility to between 1⁄4 and % mile. The captain was 
clearly uncomfortable with the approach. Weather was reported at minimums, and the 
glideslope unusable, and he was concerned about their rate of descent. “This just feels 
wrong,” he stated. After passing the final approach fix, the crew received an updated 
report that conditions were below that required for the approach. |? As they continued the 
approach, the captain stated, “This is up,’ followed by, “I’m gonna go ahead and...tell ’°em 
I missed up here,’ referring to his desire to execute a missed approach (go-around). In 
spite of the captain’s statement and concern, he allowed the first officer to continue the 
approach, despite the lack of response from the first officer in acknowledging the captain’s 
discomfort. As a tenant of CRM, communication involves both assertive statements from 
the communicator and confirmation from the receiving party. 

The crew was unable to obtain the required visuals until the moment after they 
received the “two hundred, minimums” electronic callout. At this point, the captain 
confirmed having the runway lights in sight, although immediately afterwards stated that 
he couldn’t see the runway and called for a “go-around.” The first officer, the pilot 
flying, responded that he had the end of the runway in sight and elected to continue 
with the landing. The captain questioned the first officer’s ability to see the runway, 





13 The crew confirmed with one another that they were legal to land once past the final 
approach fix, as long as the weather conditions reported prior to this point met the 
approach requirements. This is consistent with FAA and Shuttle America guidance. 
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but allowed the landing to continue. Both pilots later admitted having lost visual contact 
after descending below decision altitude.'* Given their inability to determine the 
aircraft’s location over the runway, the landing flare was unduly extended, resulting in 
touchdown approximately midpoint on the runway. By this time, there was insufficient 
runway remaining and the aircraft departed the runway’s end. 

In this case, the first officer was responsible for responding to the captain’; command by 
immediate initiation of missed approach or “go-around” procedures. Unfortunately, the 
captain’s call for a missed approach was less than assertive, > he made no attempt to reconfirm 
this decision, nor did he take command of the aircraft given the failure of the first officer to 
carry out his command. The captain is responsible for using his authority and leadership role 
to ensure the safety of flight, which involves making command decisions that err on the side 
of safety, versus conceding to the most risky attitudes or behaviors among crew members. 








™ According to FAA requirements (14 CFR 91.175) and company procedures, if 
sufficient visual references are not distinctly visible at or below the decision height 
(DH) or minimum descent altitude (MDA), execution of a missed approach is required. 
15 According to the CVR, the captain stated, “I can’t see the runway, dude, let’s go 
[around].”. This is not in accordance with AC 120-51 which states that 
crewmembers must advocate in clear, unambiguous terms, the course of action they 
feel is best, even when it involves conflict with others. This callout also did not 
comply with standard terminology for a missed approach. 
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In establishing the leadership role, the captain has the responsibility to set the tone for 
how the flight will be conducted. AC 120-51 states that the tone in the cockpit 
should be “friendly, relaxed, and supportive.” But it also points out that the crew 
must ensure that cockpit discipline is maintained, crew vigilance is not reduced and 
critical items are not missed. 

On a quiet Sunday morning in Lexington, KY, 49 people lost their lives because 
of the crew’s relaxed approach and lack of vigilance to critical items (NTSB, 2007a). 
At 6:06 am, the captain of the Comair flight 5191 aligned the Bombardier 
Regional Jet onto a runway at Lexington, KY, and turned control over to the first 
officer. As they initiated takeoff thrust, both pilots were not aware that they were on 
the wrong runway—a runway that lacked sufficient length for the aircraft to safely 
depart. 

Shortly after initiating the takeoff roll, the first officer sensed something was not 
right, commenting that the runway lights were not on: “[That] is weird with no 
lights.” “Yeah,” responded the captain. The takeoff roll continued and the aircraft 
was rapidly approaching the end of the runway. With urgency in his voice, the 
captain called that the aircraft had reached flying speed, but he did so several knots 
early, “V1 Rotate, Whooo.” The jet plowed through an open field at the runway 
end, and careened into a rising hillside. An immediate and devastating fire erupted, 
and of the 50 passengers and crew onboard, only one survived. The investigation 
revealed that, from the time they recognized the absence of lights, the crew still had 
eight seconds during which they could have safely rejected the takeoff and remained 
on the runway. 

The NTSB determined that the casual cockpit atmosphere, created by the captain, 
contributed to a reduced vigilance for the crew in conducting the pre-flight tasks. The 
captain told the first officer several times, “at your leisure,’ or “finish it up at your 
leisure,’ with regard to his performance of checklists. The casual tone began prior to 
engine start and continued throughout the pre-flight and taxi phases of flight, with 
continual non-pertinent conversation during the entire pre-flight checks. In setting an 
overly casual and relaxed tone in the cockpit, the captain set the stage for relaxed 
standards and a lack of vigilance with regard to their position on the airport. Further, he 
did not question the first officer’s statements about the lights as problematic, when an 
immediate reaction was required. 

Years ago, then- NTSB Board Member John Lauber put this in perspective: “There is 
a fine line separating a relaxed and easy atmosphere in a cockpit from a lax one where 
distractions can result in critical failures. Professionalism may be described as knowing 
the difference between the two” (Lauber, 1989). It is the captain’s leadership 
responsibility to maintain the proper balance in setting the tone, with the goal of 
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achieving only the highest level of safety standard. In aviation, there is no room for 
accepting a lower standard. 





16.1.2. Adherence to Standard Operating 
Procedures 


Another integral component of CRM is adhering to Standard Operating Procedures 
(SOPs). According to AC 120-51, “CRM training is most effective within a training 
program centered on clear, comprehensive SOPs.” Further, “A captain’s briefing should 
reaffirm established SOPs...,’ and “...sets expectations for handling deviations from 
SOPs.” Therefore, each crew member is responsible for carrying out the procedures and 
guidelines set by the company in a routine and professional manner, and any exceptions 
to this guidance should be discussed. 

The tendency to disregard company guidance has a quantifiable impact on the 
safety of operations. In 1993, Boeing analyzed 138 airline accidents that claimed over 
5,600 lives over a ten-year period. The highest accident causal factor (48%) was 
failure of the pilot flying to follow SOPs (Weener, 1993). In another study, intentional 
non-compliance was a factor in over a third (39.5%) of accidents reviewed by 
researchers using a global fatal accident database (Khatwa and Helmreich, 1999). 
Furthermore, data from the Line Operations Safety Audit (LOSA) Collaborative 
shows the importance of following SOPs (NTSB, 2007): specifically, analysis of 
LOSA data from over 6,000 line flights, where trained observers rode in cockpit 
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jump-seats and performed audits, revealed that flight crewmembers who intentionally 
deviated from SOPs were three times more likely to commit other types of errors, 
mismanage more errors, and find themselves in more undesired aircraft situations, 
compared with those flight crewmembers that did not intentionally deviate from 
procedures. !° 

The failure to adhere to SOPs is seen all too frequently in aircraft accidents. For the 
Lexington Comair accident (NTSB, 2007a), in addition to the casual atmosphere 
created by the captain, the investigation also cited the failure of the captain to follow 
company procedures for performing taxi briefings, calling for checklists, and in 
respecting the “sterile cockpit” rules prohibiting extraneous activities (e.g. conversation) 
during critical phases of flight.” The Safety Board concluded that “the flight crew’s 
noncompliance with standard operating procedures, including the captain’s abbreviated 
taxi briefing and both pilots’ non-pertinent conversation, most likely created an 
atmosphere in the cockpit that enabled the crew’s errors.” 

For Shuttle America flight 6448 (NTSB, 2008), in addition to the issues 
with leadership mentioned in the previous section, the crew failed to observe 
sterile cockpit procedures during the descent phase of flight, and elected to 
ignore reports of the unusable glideslope, instead continuing the approach to 
the lower Instrument Landing System (ILS) decision altitude. Had they 
followed SOPs in using the higher Localizer minimum descent altitude (MDA), 
the reduced visibility at that altitude would have required them to initiate 
a missed approach. !® 

The crews in these examples committed multiple deviations from SOPs, further 
supporting the premise that SOP deviations decrease safety margins and increase 
opportunities for an accident. 





16 See page 73 of the Lexington, KY, Comair Accident Report: NTSB (2007a). 

"’ Federal Aviation Regulation 121.542 prohibits flight crews from engaging in 
non-essential activities, including conversation, during critical phases of flight. 
Critical phase of flight includes taxi and operations conducted below 10,000 
feet msl. 

18 According to the NTSB report, the flight crew would have been required to 
execute a missed approach if they had been using the localizer MDA of 429 ft 
agl, with no evidence from the CVR or post-accident interviews that either 
crewmember had the runway environment in sight by that altitude. The ILS 
approach decision altitude was 227 ft. 
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16.1.3. Active Monitoring 


Active flight crew monitoring and cross-checking is another vital component of CRM. 
According to AC-120-51, the monitoring function is an essential primary flight duty: 


...to ensure the highest levels of safety, each flight crewmember must carefully monitor the 
aircraft’s flight path and systems and actively cross-check the actions of other crewmembers. 
Effective monitoring and cross-checking can be the last line of defense that prevents an 
accident because detecting an error or unsafe situation may break the chain of events leading to 
an accident. This monitoring function is always essential, but particularly so during approach 
and landing when controlled flight into terrain (CFIT) accidents are most common. 


The results of numerous studies implicate the critical responsibility of monitoring and 
cross-checking. For example, inadequate monitoring was found to be a factor in half 

(50%) of CFIT accidents (ICAO, 1994). In 1994, the NTSB conducted a safety study of 
crew-caused air carrier accidents. Of significance, 31 of the 37 reviewed accidents (84%) 
involved inadequate crew monitoring or challenging (NTSB, 1994c). As explained in 

the 1994 NTSB safety study: 


Monitoring the results of one’s actions is an important ingredient in consistent, excellent 
performance of complex tasks. In flying, self-monitoring allows a pilot to recognize 
inadequate performance, observing changes in the operational environment and take 
corrective action. Self-corrections may range from adjusting control inputs to reversing 
decisions. In [multi-crew] operations, the monitoring task is shared by two or more 
crewmembers. The flying pilot is responsible for monitoring his or her own procedures and 
control inputs, and operational redundancy is provided by the non-flying crewmember with 
the task of monitoring the flying pilot. 


A tragic example resulting from poor monitoring is the crash of a King Air A100 in 2002 
which claimed the lives of a US Senator and seven others (NTSB, 2003). The 
investigation revealed that the aircraft stalled at low altitude and the crew was unable to 
recover. The NTSB concluded that “during the later stages of the approach, the flight 
crew failed to monitor the airplane’s airspeed and allowed it to decrease to a dangerously 
low level (as low as about 50 knots below the company’s recommended approach 
airspeed) and to remain below the recommended approach airspeed for about 50 
seconds.” 

Traditional CRM courses have done an excellent job of reinforcing the 
responsibility of each crewmember to challenge others or speak up when something 
appears unsafe or unwise. Paradoxically, to challenge something, one must first detect it, 
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and detection requires effective monitoring (Sumwalt, 2004). According to Captain 
Frank Tullo, “First, we must change our approach to monitoring. Instructors must insist 
that the non-flying crewmember monitors the flier effectively. A system that grades 
monitoring must be established Good monitoring skills are not inherent in a pilot as they 
progress in their careers. Therefore, effective monitoring techniques must be trained and 
rewarded” (Tullo, 2001). Unfortunately, few CRM courses provide explicit guidance on 
how to improve monitoring, perhaps a result of the relatively recent focus of this 
element in FAA guidance materials. 

There is a continued frequency of accidents attributed to inadequate monitoring. 
For example, over half (63%) of the approach and landing accidents reviewed under the 
Flight Safety Foundation’s Approach and Landing Accident Reduction (ALAR) 
program involved poor monitoring and cross-checking (Khatwa and Helmreich, 1999). 

A recent example is the 2005 accident involving a Cessna Citation 560 at Pueblo, 
CO (NTSB, 2007c), in which the crew failed to monitor the approach speed in icing 
conditions and allowed the aircraft to slow to a speed below the normal approach 
speed.” In the accident report, the Board stated that they were “aware of recent 
accidents in which inadequate pilot monitoring was a causal or contributing factor to 
the accident and in which pilots on approach to landing failed to observe critical and 
salient cues.” Safety Recommendation A-07-13 was issued to the FAA to require that all 
pilot training programs be modified to contain modules that teach and emphasize 
monitoring skills and workload management and include opportunities to practice and 
demonstrate proficiency in these areas. 

One factor that can contribute to the crew’s failure to monitor effectively is poor 
workload management. AC 120-51 stresses that special attention be given to the level of 
workload in planning and carrying out work tasks: “Secondary operational tasks 





" AC 120-51 was updated in 2004 to highlight the role of pilot monitoring. Also, in 
2003, the FAA published Advisory Circular 120-71A, Standard Operating Procedures for 
Flight Deck Crewmember. That document stated: “It is increasingly acknowledged that it 
makes better sense to characterize pilots by what they are doing rather than by what 
they are not doing. Hence, pilot flying (PF) remains an appropriate term and is 
unchanged in this AC. But the term pilot not flying misses the point. Studies of 
crew performance, accident data, and pilots’ own experiences all point to the vital 
role of the non-flying pilot as a monitor. Hence, the term pilot monitoring (PM) is 
now widely viewed as a better term to describe that pilot.” 

20 Given the icing conditions, the crew should have maintained an even higher 
approach speed. 
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(e.g. dealing with passenger needs and communications with the company) are 
prioritized so as to allow sufficient resources for primary flight duties.” 

An FAA-funded study found that in approximately one third of the reviewed NASA 
ASRS reports, pilots failed to monitor errors, “often because they had planned their 
own workload poorly and were doing something else at a critical time” (Jentsch et al., 
1997). A 1998 NASA research project of cockpit interruptions and distractions reviewed 
107 ASRS reports to determine types of tasks that crews typically neglected at critical 
moments while attending to other tasks (Dismukes et al., 1998). Results revealed that 
almost three quarters (69%) of the neglected tasks involved either failure to monitor the 
current status or position of the aircraft, or failure to monitor the actions of the pilot 
flying or taxiing. To avoid such problems, the study suggested that crews “schedule/ 
reschedule activities to minimize conflicts, especially during critical junctions.” 

Monitoring requires active vigilance. High workload periods are defined by the 
requirement to carry out complex tasks or multiple simultaneous tasks, which can 
diminish the performance of some or all tasks (Shiffrin and Schneider, 1977; Dismukes 
et al., 1998; Loukopoulos et al., 2009). Therefore, effective monitoring, especially 
during high workload periods, requires prioritizing such that adequate cognitive 
resources are devoted to the task. 

The December 1993 crash of a Jetstream 31 at Hibbing, Minnesota, operated by 
Express II Airlines/ Northwest Airlink, illustrates how improper workload management 
can impede the critical task of monitoring (NTSB, 1994a). During the instrument 
approach, a nighttime localizer back-course instrument approach to runway 13, the 
captain became involved in directing the first officer to select the common traffic 
advisory frequency (CTAF) and to illuminate the runway lights by clicking the aircraft 
microphone seven times. While being distracted by this discussion, the crew allowed the 
aircraft to be flown into the ground 2.8 miles from the runway threshold. All 18 
occupants perished. As pointed out by the NTSB, this task should have been covered in 
the approach briefing and accomplished much earlier in the approach. 

A more recent accident illustrating the criticality of prioritizing monitoring activities 
during high workload phases of flight involved an S-76A+-+ (NTSB, 2006). During 
an overwater flight in the Gulf of Mexico, the crew established a rate of descent 
appropriate for a planned stop at a platform for refueling. Given the favorable wind 
conditions, they opted instead to fly directly to their destination platform. The crew 
changed their heading and began to discuss details associated with the change in route, 
during which time the rotorcraft slowly descended to the water at cruise airspeed. All 
ten on board perished. 

These accidents vividly illustrate the need to develop strategies to improve 
monitoring. For example, pilots should recognize flight phases where poor monitoring 
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is most problematic and strategically plan to maximize monitoring during these areas of 
vulnerability. 


16.1.4. Managing Fatigue 


According to FAA, “Recognizing and managing fatigue is one of the most important 
elements recommended for inclusion in CRM training” (FAA, 2006b). Since 1972, 
the NTSB has issued 117 fatigue-related safety recommendations in all modes of 
transportation, with 34 being aviation related. Fatigue has been on the Safety Board’s 
Most Wanted List since the inception of this list in 1990, and today’s list has seven 
aviation fatigue-related recommendations concerning air traffic control, maintenance 
and flight crew (NTSB, 2009b). Between 1993 and 2009, fatigue was associated with 
over 250 fatalities in air carrier accidents investigated by the Safety Board. 
Additionally, there have been countless general aviation accidents involving fatigue 
during that same time period. 

Accident investigations have revealed two overall issues with pilot fatigue: flight and 
duty time regulations that lack scientifically based research on pilot fatigue and sleep 
issues, and crewmembers not exercising personal responsibility to ensure they are well 
rested for work. This section will focus on the second issue: pilot responsibility. 

In 1993, a Continental Express EMB-120 was climbing through 17,000 feet when the 
aircraft stalled and plummeted out of control (NTSB, 1994b). Going through violent 
excursions, reaching 111-degree bank angle and 67 degrees nose down, the crew finally 
recovered as the airplane was descending through 5,500 feet. The crew made a forced 
landing on a closed runway, but due to systems failures induced by the upset, excessive 
speed and a late touchdown, the aircraft departed the runway’s end on landing. As causal 
factors in this accident, the NTSB cited CRM aspects mentioned previously in this 
chapter, including the crew’s failure to maintain professional cockpit discipline, and 
inattention to the flight instruments (monitoring). However, also contributing to the 
accident was fatigue induced by the flight crew’s failure to properly manage provided rest 
periods. 

Although that accident occurred in the early 1990s, the Safety Board has continued 
to see accidents where pilots have not exercised personal responsibility regarding fatigue. 
In Shuttle America flight 6448, mentioned earlier, the investigation revealed that the 
captain had slept only one of the past 32 hours. He did not advise Shuttle America of his 
fatigued state, nor did he attempt to take himself off of the trip. In post-accident 
interviews he stated that his lack of sleep affected his ability to concentrate, process 
information and to make decisions, and that he was “not at the best of his game.” 
Although the NTSB probable cause focused on the failure of the flight crew to execute 
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a missed approach, a contributing factor was the “captain’s fatigue, which affected his 
ability to effectively plan for and monitor the approach and landing.” 

The individual responsibility of each member of the crew to ensure personal fitness 
of flight prior to taking command is undisputed, regardless of the potential negative 
career implications or consequences.” However, the actions of the crew, for any 
accident, cannot be considered outside of the context of the organization in which the 
flight crew operates. 

This brings us to the second issue. As accident investigators, there is a need to impress 
upon the aviation community that in accident investigations, evaluating the role of CRM 
involves more than an evaluation of the flight crew’s actions and training—it involves 
examining the entire system in which crewmembers operate, to include management’s 
role in the development, support and consistent reinforcement of CRM concepts. 


16.2. Organizational Accountability 
and Indoctrination 


People don’t operate in a vacuum. They are affected by the system in which they live and 
work. While CRM involves the individual and the flight crew as a team, a complete 
accident investigation must go beyond merely looking at the individual at the “sharp 
end” of the aircraft. As Dan Maurino so aptly put it, “The identification of the human 
error should be considered the starting point of the investigation, not the ending point 
that has punctuated so many previous investigations” (Maurino, 1997). An 
understanding of the underlying elements that precipitated the undesirable behavior is 
needed; therefore, a complete investigation is one that also considers the system and the 
safety culture of the organization. 


16.2.1. Systems Approach to Investigation 


Human error is a causal factor in the majority of aviation accidents. But, labels such as 
pilot error” or “lack of CRM” do very little to improve safety. To 
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“human error, 





*114 CFR 61.53 and 63.19 preclude required flight crewmembers from flight duty 
while they have a known medical or physical deficiency. Although the regulations 
do not specifically cite fatigue, the FAA’s Aeronautical Information Manual (AIM) 
discusses fatigue as a factor that pilots should evaluate as part of determining their 
fitness for flight. 
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enable meaningful accident data classification and safety recommendations, the specific 
elements of CRM must be identified. By focusing on those individuals who committed 
the last error that led to the accident, valuable prevention opportunities are missed, as the 
individuals are unlikely to repeat the error. Oftentimes, the underlying factors that 
trapped those individuals still remain in the system, waiting to snare others. As stated in 
the Columbia Space Shuttle Accident Investigation Report (NASA, 2003): 


Many accident investigations make the same mistake in defining causes. They identify the 
widget that broke or malfunctioned, then locate the person most closely connected with the 
technical failure: the engineer who miscalculated an analysis, the operator who missed signals 
or pulled the wrong switches, the supervisor who failed to listen, or the manager who made 
bad decisions. When causal chains are limited to technical flaws and individual failures, the 
ensuing responses aimed at preventing a similar event in the future are equally limited: they 
aim to fix the technical problem and replace or retrain the individual responsible. Such 
corrections lead to a misguided and potentially disastrous belief that the underlying problem 
has been solved. 


As stated, prevention of accidents is the goal of accident investigation (FAA, 2004; 
ICAO, 2001). Given the strong role of organizational factors in many accidents, 
investigations that only document the obvious human errors may overlook systemic 
factors that would enable the prevention of future accidents. 

The following two accident examples illustrate the change in perspective of the 
Safety Board over a 13-year period regarding the role of the organization in promoting 
safety, and in promoting CRM behaviors. In February 1979, a Nord 262 air carrier 
crashed at Clarksburg, WV. The NTSB determined the probable cause as “the captain’s 
decision to takeoff with snow on the aircraft’s wings and empennage surfaces...” 
(NTSB, 1979a). The report focused solely on the actions, errors and decisions of the 
flight crew, in effect, the inadequate CRM of the crew. Moreover, in the 13 years that 
succeeded this accident, ten air carrier accidents worldwide were attributed to the same 
situation—takeoff attempts with snow and ice on wings. 

In March 1992, a Fokker F28 crashed during attempted takeoff from New York’s 
LaGuardia Airport, with snow and ice on the wings (operated by the same airline as in 
the aforementioned accident example). In contrast to the 10 previous investigations, this 
investigation evaluated the crew’s performance and decisions as well as the system in 
which the flight crew operated. As part of the probable cause, the accident report cited 
the failure of the airline industry and the FAA to provide flight crews with procedures, 
requirements and criteria compatible with departure delays on conditions conducive to 
airframe icing (NTSB, 1993). Although the NTSB also cited the role of the crew’s 
performance and decisions, by citing the system as the foremost cause, the catalyst for 
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change was set. The crew’s errors were placed in context, resulting in more broad- 
reaching recommendations aimed at correcting the system. 

The systemic investigative approach that surrounded this accident directly 
influenced widespread improvements to FAA regulations pertaining to takeoff in icing 
conditions, as well as massive overhauls to air carrier operating procedures. Had the 
investigation merely pointed the finger of blame at the flight crew, little, if any, true 
remedial actions would have emerged from this accident investigation (Sumwalt, 1995). 


16.2.2. Organizational Commitment to Safety 


Through over four decades of accident investigation, the Safety Board has found that, 
oftentimes, the attitude of the organization towards safety is the most common link to 
its safety record. For pilots to adopt and consistently carry out the principles of CRM, 
there must be a strong and resounding management commitment to safety and CRM, 
which a statement embedded in the organization’s mission statement simply can 
achieve. 

The concept of the Safety Management System (SMS), developed over the past 
decade, provides a solid framework for the organizational behavioral components 
required to achieve and maintain a higher level of safety (FAA, 2006a; ICAO, 2009), and 
therefore provides an outline for focus areas of accident investigation in evaluating the 
role of the organization in promoting CRM behaviors and attitudes. One component of 
this system-wide commitment is having policies and procedures that support the crew’s 
ability to make decisions geared towards the goal of safety. According to AC 120-51, 
“Flight operations manuals and training manuals should embrace CRM concepts by 
providing crews with necessary policy and procedures guidance centered on clear, 
comprehensive SOPs.” 

The NTSB has recently cited the role of the company in accidents attributed to 
elements of CRM, to include inadequate decision-making and leadership. Returning 
to Southwest Airlines flight 1248 (NTSB, 2007d), the crew spent considerable time 
during the en route flight phase in making their decision to land. Company guidance 
indicated that they were authorized to land with any positive stopping margin as 
displayed by the on-board performance computer. However, the assumptions 
underlying those calculations were not displayed with the output,” and the written 





*° Critical assumptions were not presented on the display: the use of a lower tailwind 
component than that which was entered by the crew, that thrust reverse was included in 
the calculations, and that there was no safety margin. 
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guidance was inconsistent from one section of the flight manual to the next. Not only 
did the crew feel compelled to land based on the stated guidelines, but they were not 
aware of the risks they were assuming. Further, new company guidance indicated 
that, in certain weather conditions, the crew was required to use autobrakes. The 
captain felt uncomfortable in making the landing and in using the autobrakes for the 
first time in winter conditions, but in this case, he was forced to choose between 
following SOPs and the responsibility to make a safe landing. This conflict made it all 
too easy for him to cede his leadership role and accept the perspective of his first 
officer. Unfortunately, company policy, as was written at that time, did not support 
his making a safe landing decision. In the accident report, although the NTSB cited 
the decision of the crew not to divert, they also recognized the role of the company 
in influencing the captain’s decision to land: 


Contributing to the accident were Southwest Airlines’ 1) failure to provide its pilots with 
clear and consistent guidance and training regarding company policies and procedures related 
to arrival landing distance calculations; 2) programming and design of its on board 
performance computer, which did not present inherent assumptions in the program critical to 
pilot decision-making; 3) plan to implement new autobrake procedures without a 
familiarization period; and 4) failure to include a margin of safety in the arrival assessment 
to account for operational uncertainties. 


Another example of the role of the organization in supporting CRM principles is 
Shuttle America flight 6448 (NTSB, 2008). In addition to the findings mentioned 
previously, the investigation also revealed issues related to the company’s fatigue policy. 
Recall that in this accident, the captain failed to assert his authority in enforcing the 
go-around call, and failed to follow SOPs in doing so. Another aspect of this accident is 
the role of the captain’s fatigue for this flight. According to company policy, the captain 
had accumulated an excessive number of sick calls. Within weeks prior to this flight, 
Shuttle America had notified the captain that additional sick calls would result in formal 
discipline, up to and including termination. According to post-accident interviews the 
captain believed he would be terminated if he called in “fatigued” for this flight. 
Therefore, although the final report cited the role of the crew in their failure to execute 
a go-around, a contributing factor was Shuttle America’s “failure to administer an 
attendance policy that permitted flight crewmembers to call in as fatigued without fear 
of reprisals.” 

Leaders of a company have an obligation to provide well thought-out procedures. 
And they also have a job to ensure that people follow them. For example, in the 2007 
accident involving NASCAR in Sanford, FL (NTSB, 2009a), the company did not 


The Accident Investigator’s Perspective - Chapter 16 419 


support strict adherence to SOPs and regulations regarding maintenance. The NTSB 
determined the probable cause as: 


...actions and decisions by [NASCAR‘°s] corporate aviation division’s management and 
maintenance personnel to allow the accident airplane to be released for flight with a known 
and unresolved discrepancy, and the accident pilots’ decision to operate the airplane with that 
known discrepancy, a discrepancy that likely resulted in an in-flight fire. 


As evidenced by this example, promoting safety through policies and procedures is yet 
one of the many components that comprise a healthy SMS. Although it exhibits the 
company’s attitude towards safety and promoting cultural assimilation and adoption of 
CRM principles, full management support of these polices is critical. If an 
organization places a meaningful emphasis on the principles of CRM, the resulting 
increase in safety will be obvious and measurable: a reduction in errors and an increase 
in safety. 


16.3. Summary 


In this chapter we advocate a multi-faceted approach to improve the effect of CRM on 
crew performance. In the first section we identified several aspects of CRM that, having 
appeared with some frequency in recent accidents, warrant further emphasis in the 
broader conceptualization of CRM. These aspects include the role of individual 
responsibility in taking leadership, in adhering to standard operating procedures, in 
active monitoring and in making responsible decisions in managing fatigue. Set within 
the context of the team environment, these individual roles and responsibilities are 
integral elements of CRM as defined by AC 120-51. 

The second component, equally important, is the role of the organization in being 
accountable for indoctrinating the principles of CRM throughout each level of the 
system. For the accident investigator, this perspective urges a systems safety approach to 
examine the contribution of CRM in each accident, through evaluating failures and 
successes at the individual and crew level, as well as the role of management in the 
indoctrination and support of CRM principles system-wide. As stated by Anthony 
Hopkins (2005), “Errors by frontline operators are hardly an adequate explanation for 
why something went wrong.” 

This multi-faceted approach is set in a broader context, one which embraces the 
perspective advocated by Helmreich, Merritt and Wilhelm (1999), that the focus of 
CRM should be to reduce errors and increase safety, not simply to increase the 
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behavioral components addressed in AC 120-51. Firm decisions, policies and 
procedures should be based on this goal, at both the individual and organizational level. 
With the ultimate goal of safety in mind, the front-line employee, their immediate 
team, the operator, the regulating agency, and the accident investigator can work 
together to effect meaningful change in reducing accidents and incidents resulting from 
the failure of the crew to carry out the behaviors and principles embodied by CRM. 
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Introduction 


The beginning of the Jet-age was a very exciting time for commercial aviation. 
Advanced technology jet aircraft flew faster, and at higher altitudes, not to mention the 
remarkable improvements to passenger comfort. With this new era came additional 
risks, some of which were not obvious to the aircraft designers and engineers. During 
the transition from propeller-driven aircraft to jets, it was apparent that there was a need 
to improve safety. Fatal accidents were occurring all too frequently and were highly 
publicized. Causal factors included both technical and mechanical issues. In the majority 
of commercial accidents, the primary cause was listed as “pilot error.’ For the aviation 
industry to survive as an acceptable mode of transportation, these issues and the risks 
they represented would need to be mitigated. 

Advancements in aircraft equipment design played a large role in reducing the 
accident rate and in many cases also addressed human error. The development and 
implementation of systems such as the Terrain Collision Avoidance System (TCAS), 
Enhanced Ground Proximity Warning System (EGPWS) and the New Generation 
Flight Deck with map displays provided a technical solution to address many of the 
threats flight crews face everyday. These technological solutions also helped flight crews 
to eliminate a degree of human error and better manage the errors that occurred. 
Additionally, the advent of the flight simulator and associated improvements in training 
were also effective strategies in reducing the accident rate. Technological solutions were 
not enough. As the numbers of flights continue to increase, the accident rate will require 
further reduction, and therefore there will be a need to focus on the role of human 
resources if aviation is going to make any additional safety improvements. 

In 1978 a commercial airline crashed when the crew mismanaged an aircraft mal- 
function, lost situation awareness and ran out of fuel. What brought this aircraft down 
was the inability of the crew to work together and manage the additional workload 
associated with the malfunction. Following this accident, in the early 1980s, the aviation 
industry began to focus on human factors training with special emphasis on leadership 
and decision-making. Those first courses had titles such as Crew Coordination 
Concepts (CCC) and Crew Leadership Resource (CLR). These early programs 
targeted only flight crews and were designed using corporate-style management 
training. Using terms such as the “Theory of the Situation” did not lead to complete 
buy-in by flight deck crews. Many of the professional pilots who attended these courses 
referred to the training as “charm” school. Without crew buy-in, the effectiveness of this 
training was questionable. A change in course was eventually made and the industry 
began a journey that included the development of Crew Resource Management 
(CRM) programs which were initially aimed at pilots and then eventually included flight 
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attendants, mechanics and dispatchers. These CRM programs formed the foundation 
for the building of a new safety culture within the industry. 

Early versions of CRM played an important role in reducing the accident rate. CRM 
provided the initial tools for crews to use in developing leadership and team building 
skills. In the dynamic environment of the aviation industry, flight crewmembers not only 
required technical skills but also leadership and followership skills! (Petit and Dunlap, 
1997). The International Civil Aviation Organization (ICAO) captured this notion in 
“Fundamental human factor concepts”? (ICAO, 1998): “Leadership skills should be 
developed for all (crewmembers) through proper training; such training is essential in 
aircraft operations where junior members are sometimes called upon to adopt a lead- 
ership role throughout the normal performance of their duties.” The importance of 
expanding CRM to include employees outside of the flight deck was critical in 
developing the team to better manage threats throughout the entire operation. Even 
with these efforts, there was still an unacceptable accident rate and there was room for 
additional “risk” reduction. In the mid 1990s, it was estimated that with the projected 
growth in the industry and no further reduction in the worldwide accident rate, the 
industry would eventually suffer a hull loss every week. 


17.1. Threat and Error Management 
(TEM)—a Process That Works 


A dramatic change in how airlines collect and use safety data to address system threats 
and to design more effective human factors courses was introduced by Professor Robert 
Helmreich and the University of Texas Aerospace Human Factors Research Project. In 
1996, Continental Airlines working with Professor Helmreich’s team conducted 

a system-wide Line Operation Safety Audit (LOSA). LOSA is a non-jeopardy obser- 

vation of line crews during normal operations. LOSA is designed to identify overt and 
latent threats, crew errors, and aggregate crew performance on how threats and errors 
are managed. The analysis of data on the management of threats and errors would enable 
organizations to better determine effective countermeasures. LOSA also identifies the 
links between crew errors, threats and additional errors, as they become the precursors to 
accidents and incidents. For LOSA and its subsequent training to succeed, the data and 
its analysis would need to be presented to senior management. Management teams were 
educated on the value of using this non-jeopardy reporting system to target safety 





" “Cockpit leadership and followership skills” by Pettit and Dunlap, March 1997. 
* “Fundamental human factor concepts” by ICAO, 1998. 
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change. Management also needed to gain an understanding that it is more important to 
use LOSA data to affect system changes than policing individual crews or pilots. An 
additional benefit of LOSA is the policy to brief the senior leadership on the LOSA 
results. When Continental’s chief operating officer (COO) reviewed the results of the 
airline’s 1996 LOSA, he turned red in the face and told his managers to “fix it’ Based on 
the 1996 LOSA results, it was evident that flight operations at Continental needed 
improvement in areas such as checklist usage and stabilized approaches. 

After each LOSA, Continental also formed committees to look at specific areas of 
concern. Following the 1996 LOSA, the Stable Approach, Checklist and Leadership 
committees were formed. These committees consisted of both instructor and line pilots 
and they were tasked with reviewing the LOSA results and developing appropriate safety 
changes. These changes targeted policy and procedures as well as crew training. These 
recommendations were reviewed and approved by the Flight Standards and Training 
department and then implemented. 

Working with the University of Texas Aerospace Human Factors Research Project, 
Continental developed an Error Management (EM) course that differed from previous 
CRM programs. To accomplish this new human factors training and effectively change 
the pilot “culture?” Continental focused on crew error and the countermeasures 
necessary to avoid, trap and mitigate errors before they become consequential. The 
decision was made to deliver the training using pilot facilitators and to limit the classes to 
a maximum of 20 pilots. Selected “line” pilots attended a two-day course on facilitation 
to prepare them for effective delivery of the course material. The use of pilots to conduct 
the classes adds credibility and limiting the class size helped ensure total participation. 
Pilots attending the new human factors training course would need to accept that errors 
will occur as there are no perfect flights. One of the opening questions during the EM 
training program was, “Have you ever had the perfect flight where no errors were 
made?” After several classes of never having a positive response, one pilot raised his hand. 
When asked about his perfect flight, he responded, “My trip was bought by Flight 
Operations and I got to stay home.” 

Figure 17.1 represents the initial Error Management model. Errors needed to be 
trapped or mitigated before there was a consequence. The EM course introduced Resist 
items such as Standard Operating Procedures (SOPs), checklists, Air Traffic Control 
(ATC), and training. All these are important guards against possible errors. The Resist 
area is where most errors were trapped. Yet even with these methods for resisting error, 
they would occasionally slip through. An example of an error would be when a flight 
crew receives an altitude change and reads back the wrong altitude. Normally, ATC 
would confirm the readback and trap the mistake. Occasionally, ATC misses the 
incorrect readback and the error still has possible consequences. There is still a chance to 
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Figure 17.1 The initial Error Management model 
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catch the error through the use of CRM countermeasure skills (the Resolve component 
of the model) which are effective in managing error. For instance, using leadership skills 
to foster an open cockpit where good communication and cross-checking occur 
without barriers would be another countermeasure to trap this type of error. When the 
altitude change is received, both pilots must verify the new altitude, and in an open 
cockpit doubts are expressed and this leads to additional verification of the clearance 
with ATC. 

CRM skills such as communication, workload management and team building are 
the very basis of a good EM program. Effective communication and decision-making 
represent the qualities of good leadership and are expected in all crewmembers. Training 
on decision-making focused on the captain’s effectiveness to form the team in order to 
improve the decision-making process. The notion that what is right is more important 
than who is right was key in team development and effective decision-making. The 
process of trapping errors through the Resist and Resolve items was established. The 
CRM skills that have been developed in earlier courses are the “heart and soul” of 
Resolve and are essential to the success of the EM process. Pilots now understood that 
a safe flight was not about total error avoidance but about how errors that do occur are 
managed by the crew. 

One of the more difficult issues in establishing acceptance of an EM program is 
training the check airmen to accept that crew error will occur and to reward effective 
EM when it is present. CRM markers and error management must also be imbedded 
in training courses where crews are held accountable not only for their technical 
skills but also their skills at using CRM markers as effective error countermeasures. 
Pilots will gain an understanding that “pilot error” has never caused an accident. 
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What leads to the accident is the fact that the crew failed to manage the error prior 
to a consequence. 

In 2000, Continental conducted its second major LOSA to see what effect, if any, the 
EM program had on the operation. The Line Operational Safety Audit (LOSA) 2000 
data analysis, when compared to the 1996 LOSA data, showed that Continental Airlines 
pilots not only accepted the principles of EM but had incorporated them into their 
everyday operation. LOSA 2000 showed a sizeable improvement in the areas of checklist 
usage, a 70% reduction in non-conforming approaches (not meeting company stable 
approach criteria) and an increase in overall crew performance. The “culture” at 
Continental had taken a turn in the right direction. 

LOSA 2000 data also led to the development of an addition to EM that seems to be 
the logical next step in the CRM journey. LOSA observers noted that effective crews 
were not only good error managers but more importantly were good “threat” managers. 
A threat is an event or error that occurs outside the influence of the flight crew and 
increases the complexity of the operation. Threats require crew attention and 
management if safety margins are to be maintained. In managing threats, crews can 
reduce the errors that occur in flight. “Threat and Error Management” (TEM) describes 
the threats we face everyday and reviews the strategies or countermeasures we need to 
counter them (Figure 17.2). 

LOSA 2000 observers were trained to observe and record the threats crews faced and 
how they were managed or mismanaged. There were external threats (weather, main- 
tenance, passenger problems, operational pressures, distractions/interruptions, ATC 
errors, language/communications problems, etc.) that were not crew errors but came 
from external sources and increased the potential for errors by the crew, if not managed 
properly. What surfaced during these observations were the strategies good pilots use to 
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effectively manage threats. These strategies, proactive in nature, were sometimes personal 
techniques that pilots have developed over time in order to effectively operate in today’s 
complex environment. Many strategies that are common among crewmembers and used 
routinely have developed into Standard Operating Procedures (SOPs). An example 
would be the use of a pre-departure briefing to review a power loss on takeoff. This 
briefing prepares the crew to make a better decision, should a loss of power occur when 
risk is high and decision time is minimal. Crews that effectively use strategies manage 
external threats successfully and reduce crew errors associated with these threats. Because 
of its effectiveness in identifying threats and preparing crews to manage both threats and 
errors, briefings have become a standard procedure in the industry. 

In reviewing several accidents and incidents over the last several years, it is apparent 
that, in each case, there were several threats not properly managed that preceded each 
event. There were also crew errors that also played a significant role in the mishap. When 
crews successfully recognize and acknowledge threats, they are in a better position to 
manage that threat, so it does not become consequential. Accident/incident crews 
typically do not recognize all the threats, or their severity, and the threats accumulate, 
which invites crew error. Crews are most vulnerable when they acquire several threats 
and have employed no strategies to manage them. The accumulation of threats, either not 
addressed or identified, places the crew at the edge of the operating envelope where time 
and options are limited. To effectively manage threats, they must be identified, then 
assessed and then countered. Identification of threats comes through many system 
alerting methods, such as the aircraft system alerting lights, bells, horns, voice and devices 
such as GPWS, TCAS, wind shear alerting systems, etc. Those devices may also lead the 
crew towards the best course of action to counter the threat. However, not all systems and 
devices detect, assess and counter all threats. Effective use of CRM countermeasures 
provides primary threat detection and management for threats not having system alerts 
and provides backup for those that do. Abundant examples are available which show that 
failure to employ CRM countermeasures was the last option for detecting a threat and 
avert a potential incident. One of the primary countermeasures to managing threats 
effectively is flight deck leadership. The attributes of flight deck leadership (i.e. setting the 
example, planning ahead, initiative, fostering communications, etc.) are the foundation 
for effective strategies to manage the threats crews face every day. 

An example of a threat flight crews encounter often would be a thunderstorm on 
the departure end of the runway as a crew prepares to take off. After identifying the 
threat, the initial strategy would be to turn on the radar and evaluate all takeoff options. 
The airline most likely has a strategy in place which includes guidelines on when 
a takeoff is not recommended based on the severity of the storm. The error might occur 
when the crew fails to turn on the radar and accepts the takeoff clearance. There is an 
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opportunity for the error to be trapped by ATC (Resist) through a warning of weather 
on the departure path. If ATC fails to issue this alert, there is the possibility of trapping 
the error when one of the crewmembers expresses doubts about being able to depart and 
the other crewmember accepts that input and cancels the takeoff clearance (Resist). 

Other threats are not so obvious and crews must develop good threat identification skills 
through training and open communication with other members of the operations team. 

Effective threat management reduces the complexity of the operating environment, 
decreasing the potential of crew error. Strategies need to be employed consistently so all 
threats can be more easily recognized and managed. Interactive communications, 
vigilance, monitoring and cross-checking are full-time strategies that assist crews in 
identifying most threats. To be effective, these countermeasures should be reviewed and 
deployed proactively instead of waiting for the threat to appear. In non-academic terms 
it means, “Get your stuff together before the stuff hits the fan,’ or words to that effect. 

The human factors class is not the only reason Continental Airlines saw improve- 
ment in the LOSA 2000 data and the following LOSA observations in 2004 and 2008. 
The process of changing a culture is not that simple. In any company, this change begins 
at the top and works its way down. When senior management approves and funds for an 
additional day of human factors training for all the pilots, this represents a huge 
commitment to the TEM concepts. In 1997, it might have been easy to say that several 
incidents at Continental motivated the approval for the EM class. In the subsequent 
years, the motivation for the TEM class was an understanding of and belief in the LOSA 
process. The approval came because things were going well at Continental and TEM was 
seen as a way to counter such threats as complacency and keeping the crews on the path 
towards operational excellence. Another extremely important reason TEM 1s effective is 
the commitment to its principles by check airmen. When TEM is embedded into 
training and line operations and is fully supported by the check airmen, it then has the 
potential to make a positive change in the way an airline operates. If the check airmen do 
not buy into and practice the principles of TEM, then the program will not succeed. 
Lastly, the most important reason TEM succeeded at Continental was because the line 
pilots applied it to their everyday operation. Pilots apply TEM and believe it can give 
them that added margin of safety in a complex operating environment. 

By discussing what potential threats they may face prior to each flight and then 
determining what strategies they need to have in place to counter each threat, crews will 
be better prepared to properly handle the challenges of each flight. Developing a better 
understanding and ability to assess threats and using various strategies, both personal and 
corporate, will help a crew to reduce the errors they commit and enhance the EM 
process for trapping errors that do occur. It was noted in LOSA that many crews were 
performing well during normal operations and that the majority of problems occurred 
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when the crew had several challenges thrown at them. One of the methods outstanding 
crews used was to review the “What ifs” early. They were therefore better prepared to 
handle any situation with few problems. Briefing taxi procedures prior to pushback and 
landing, thorough review of special engine out procedures, crew briefs that included 
a discussion of contingency procedures and good workload management are just a few of 
the “best practices” outstanding crews used to counter existing and potential threats. 

The environment we operate in has only become more complex over the last few 
years and will continue to present our pilots with an increased number of challenges. 
Every pilot group must “raise the bar” and accept that outstanding performance today 
will only be standard in the future. They must continually improve their TEM skills to 
maintain a safe performance level. TEM training is one of the means pilots can use to 
reach an improved level of performance that will enable them to deal with the increased 
challenges of maintaining a safe operation. The idea behind all CRM courses is to define 
the “best practices” in applying TEM countermeasures to reduce or eliminate the 
consequence of threats and errors, which are the precursors of accidents and incidents. 
Safer operations can be obtained by imbedding the best practices of our pilots into our 
training and everyday operation. Pilots learn many of their positive traits from the 
sharing of ideas and experiences with their peers and then applying them to their own 
operational philosophy. The TEM course is a means by which the pilots can prepare for 
the future and be part of the team successfully building a TEM culture. 


17.2. Develop TEM Outside of Flight 
Operations 


Developing TEM outside of flight operations will help a company to develop a pos- 
itive safety culture throughout their operation. Developing TEM outside of flight 
operations has its challenges but with the right program development team it is 
achievable. When developing TEM course material for each division, it is imperative 
that each division’s course developers fully understand the TEM concept. The next step 
is to develop the TEM terms, concepts and case studies that have direct application to 
their division personnel. 

The same processes used to reduce risk and improve flight operational performance 
will be equally effective in other divisions. This entire process is a natural extension of 
a safety management program. Ramp personnel, when attending TEM training, will 
easily grasp the concept of threats they face such as ground equipment in the taxi 
clearance zone, ice on the ramp or an unfamiliar gate area. They also understand the need 
to have a plan on what strategies to use to manage the threats and reduce the errors that do 
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occur. The same human factors skills used by flight operations will work for ramp 
personnel when they are managing threats and errors. The understanding of the TEM 
process and its application has led to significant reductions in both ground damages and 
employee injuries at one carrier. Additionally, non-jeopardy LOSA-type observations 
provide a wealth of data for course development, changes to policy and procedures and 
measuring improvements following the implementation of safety enhancements. 


17.3. TEM as an Investigation Tool 


Safety investigations can also use the TEM process for a more effective tool in root cause 
analysis. Triggered by an event or a near-miss the safety investigator can use the TEM 
process to conduct a thorough analysis of what occurred. The initial step is to list the 
threats and errors and whether they were managed or mismanaged. Also linking errors 
to threats will many times identify latent (systemic) threats that otherwise would have 
been overlooked. 

The fact a threat was mismanaged might indicate that it was never properly iden- 
tified, or strategies to manage that particular threat were not employed. This could help 
focus training on improving the skills in this area. The same process would be utilized for 
errors. A mismanaged threat often leads to an error and a mismanaged error can lead to 
an event or near-miss. Since the error was not trapped by either the safety hardware and 
policies put in place (Resist) or by the employee through use of their CRM skills 
(resolve), the analysis will lead to better corrective action. This type of investigative 
analysis will help identify needed changes in training, safety hardware, policy and 
procedures, and human factors training. 

NTSB Board Member Robert Sumwalt has stated that the presence of a “safety 
culture” in a company is when your employees do the right thing even when no one is 
watching. The LOSA and TEM programs are an integral part of a company’s Safety 
Management System (SMS) and are key elements in establishing a safety culture. TEM is 
a process that enables employees to better identify and manage threats and errors in their 
everyday life. By doing so, they no longer think of “‘safety first” but rather they establish 
safety as a core value. 
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Introduction 


Crew Resource Management proliferated from the USA. Its journeys outside of the 
USA were difficult in comparison to Clark Kent’s evolution as Ubermensch.! CRM 
migration outside the USA varied—some were welcomed with approval, punctuated 
with ease in adapting to the airline’s culture and operations, a few rejected as outright 
persona non grata and others fraught with booby traps unknowingly triggered many 
years after being implemented. Fortunately, most CRM concepts traveled painlessly 
through cultures. This chapter will examine how CRM diffused outside the USA in 
Middle Eastern, Asian and Latin American groups where front-end flight crew literature 
may not have been as readily available in the public domain. 

An undertaking to fully canvass CRM experiences outside the USA would have 
involved a comprehensive survey of global scope. Instead, to gather a cross-section of 
these experiences, this exploration took the form of probing questions which were sent 
to South American, Asian and Middle Eastern airline CRM developers and managers. 
Their personal experiences are in this chapter. All of the contributors are pilots who 
have, on the average, eight to 15 years of CRM design and delivery experience. This 
chapter is best treated as a journal of “conversations” from its contributors and reflec- 
tions of the author about the voyages of CRM outside the USA. 

We extend to our contributors our profuse thanks for building this short journal of 
cultural experiences on CRM migration outside the USA. 


18.1. Probing Questions 


Three broad areas of CRM influences were selected as important ingredients of the 
chapter. The first was about perceptions on CRM successes in relation to local 
operations. The second was about the impact of Threat and Error Management (TEM) 
on CRM-—it was important to collect opinions about how TEM was understood as 
a framework and its relationship with CRM. And the third was a “crystal ball” gazing 
exercise at the future of CRM in the respective contributors’ countries. 

Responses to the questions were collected and form the bulk of information in this 
chapter. Questions included the following: 

In your experience and opinion: 


1. What are the successes of CRM? Responses to this question included: 


m the influence of CRM on flight operations, training and simulation; 





TE 
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m the application of CRM concepts by the airline’s instructor and check pilots 
and 


m extending CRM to maintenance and engineering, inflight services and other 
departments; 


2. What are your airline’s experiences about using Threat and Error Management 
(TEM) and/or Line Operations Safety Audit (LOSA)-type data collection in 
CRM training? 


3. What is the future of CRM in your country? 


Perhaps the caveat to be understood from the beginning is that CRM was made in 
the USA. That said, its continued efficacy can only be as good as how it is shaped to 
adapt to local conditions; a country’s language and culture exerts power upon how 
CRM is adapted and changes the manner in which it is taught, used and measured. 
What is proudly American may be durable, but may not be enduring without 
adaptation. 

On the other hand, the tendency toward producing CRM adaptations to allow for 
cultural differences, while justified, opens the risk of misinterpretation. To address 
faithfulness-to-concept while maintaining cultural diversity, the significant role of the 
International Civil Aviation Organization (ICAO) as a guide should be reinforced. 

Verite en-deça des Pyrenees, erreur au-deta (Pascal, 1623-62) “there are truths on this side 
of the Pyrenees that are falsehoods on the other.” The prophecy of 16th century 
mathematician and philosopher Blaise Pascal is perhaps the best starting point on which 
to base these conversations about CRM outside the USA. These conversations present 
a starting point about the successes, adaptations and outlook of CRM travels beyond the 
USA. 


18.2. The Successes of CRM 


The dominant responses about CRM successes in non-US programs point to a few 
interesting areas which were focused on the novel methodology that was used to deliver 
the training. 


18.2.1. The Use of Line Pilots as CRM Facilitators 


Behavioral training is traditionally the domain of a trained facilitator. In fact, some 
incipient CRM programs were run either by training consultants or by retired 
flight crew. The downside of course was that the program loses a degree of 
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credibility and relevance because its agents are distant from current operational 
reality: 


Using pilots as facilitators was a welcome move, and assigning line pilots to facilitate was 
even more revolutionary. It added a substantial impact on the efficacy of the program. 


(Khoo, 2009) 


There are, however, downsides in employing this technique. In very hierarchical 
organizations, the introduction of CRM in its early days was usually perceived as 
a management maneuver and therefore not to be trusted. It was like a figurative 
management Trojan horse. 

The situation therefore puts line pilot facilitators into deeper scrutiny and, at times, 
in dissonance. Fortunately, the operational context of the program and its focus on 
managing human error almost always exonerates the line pilot facilitator. Experience has 
shown that changes expected of the airline that may arise from the conduct of CRM 
proves more diffcult to address, thus diluting both the facilitator’s work and the intent of 
the program. In one such occasion, a facilitator once said (indicating that nothing was 
being done by management to address the issues raised during the program): 


The letdown is not that management gives the same response to our questions, but rather the 
same questions seem to be endlessly raised. 


18.2.2. The Ascendance of the Co-Pilot 


The Dutch scientist, Professor Geert Hofstede, whose work on understanding the 
elements of national culture has often been used in aviation research when comparative 
analysis of flight deck culture and its dynamics of authority is needed. High power/ 
distance cultures like China, Latin America and some Asian countries stress that respect 
for rank, elders and leaders is paramount. It is not surprising that responses about the 
acknowledgment of the co-pilot’s role in flight safety was a hallmark success in the 
implementation of CRM: 


In line operations, the biggest beneficiary of all is the co-pilot. He is empowered to be more 
assertive in the name of CRM. I do feel that CRM has helped to swing the trans-authority 
gradient in the favour of the first officer. In the pre-CRM days, I felt the TAG (Trans 
Authority Gradient) was rather steep. 

(Kampli, 2009) 


From a CRM conceptual view, “empowering” the first officer, while a welcome 
by-product, was never the intent. Again, the management of human error has 
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primacy—that for authority (the captain) to better manage potential threats and 
errors, co-pilots need to be assured that their inputs are important and 
encouraged. 

Reshaping the view of pilots being invincible was also a positive outcome in that 
cooperation balanced off authoritarian leadership. Because of this, the inevitability of 
pilots making mistakes became more understood rather than shunned. As Captain 
Alejandro Camelo writes: 


Realising the need of teamwork, being aware of our own limitations and those of the other 
members of the crew has improved decision making and harmony in the working environment. 
The crew performed more professionally in their different roles. 

(Camelo, 2009) 


18.2.3. Introducing a New Vocabulary 


In many instances a new concept may not necessarily be “new” because 
linguistically it may have been labeled by a less efficacious or less memorable term 
but by all means part of a repertoire of actions. For instance, decision-making, 
communication, teamwork, cooperation and briefings were standard concepts which 
were performed by rote and habit as a function of flying operations. However, 
when these concepts were amalgamated to form clusters of what is now called 
Crew Performance Markers or NOTECHs, the meaning of those words was 
enhanced and improved. This is best captured in the words of Captain Alejandro 
Covello from Argentina: 


Another is cultural change; it introduced a new vocabulary which was integrated into the 
manuals, CRM set up a new and fundamental concept legitimated by operational personnel 
linking various areas. It is the learning of affection through experience that leads to 
attitudinal change. 

(Covello, 2009) 


Because the disparate words were associated and grouped to form an entirely 
systematic concept such as clustering into Crew Performance Markers, processes such as 
attitude and performance measurement were introduced. A similar response was 
received along these lines from Captain Alejandro Camelo: 


The major goal is to achieve quality training based on a continuous improvement concept in 
which instruction is assessed and checked at every stage, thanks to an analysis that generates 
real knowledge of the instruction level in the company. 

(Camelo, 2009) 
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18.3. CRM Influence on Check, Training and 
Simulation 


The more the questions veer toward the impact of CRM on established training, 
simulation and checking systems in flying operations, the less encouraging the responses. 
There is a large indication that this may have been caused by the lack of support for 
check and instructor pilots to be adequately trained on CRM concepts. As one of the 
responses reveals: 


A lot of instructors have a poor idea of CRM principles. Most of them associate CRM 
mainly to teamwork and communication aspects only. 
(Kampli, 2009) 


A response from a different airline indicated their difficulty in having instructor 
pilots attend CRM training, where tensions and connotations about “old school” 
instructors as not being receptive to CRM training were rife. This will be a persistent 
problem in a situation where the CRM program is personified as a line pilot gathering in 
which inadequate training is given to instructor pilots about CRM and its error 
management goals: 


A problem we experienced was when we started organising workshops; we started off with 
a lack of definition regarding the scope of Human Factors and CRM. Therefore CRM and 
operations training became wide apart. 


(Covello, 2009) 


CRM has many linkages with other airline training or competence systems, which if 
not aligned may prove at the very least confusing. For instance, CRM research has 
continuously acknowledged that lack of assertiveness by first officers cannot be fixed by 
giving more assertiveness training. Rather, the solution is to ensure an atmosphere in the 
flight deck where open communication is encouraged. Largely, this tone is set by the 
captain. 

One airline provided a dilemma where legacy programs may need to be reviewed in 
the context of encouraging assertion by first officers. The airline maintains a confidential 
competence reporting system that is embedded in their command training. This 
reporting system involves instructors to “check” and submit confidential reports about 
first officers in assessing their overall command ability and preparedness. Generally, these 
“checks” are not announced. It is by all accounts a jeopardy system which can have 
a drastic impact on one’s promotion as a captain. As such, when first officers fly with 
instructors, they tend to be on their guard, closely watching what they do and say since 
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there is a possibility that they could be on “check.” Because this system has a stronger 
impact on one’s career with deleterious effects if undesired behaviors (such as being 
more straightforward) are demonstrated, silence wins in favor of assertiveness. 

CRM that remains in the classroom has a greater propensity to decay rather than one 
that is continuously reinforced in the simulator. One of the responses reported that, 
initially, the inclusion of CRM Crew Performance Markers was useful for simulation 
training. It provided the context by which scenarios were designed and debriefed. 
However, anchoring merely on the markers soon lost ground because the markers were 
repeated and learning soon became dissociated from the scenario itself. Instructor input 
or curriculum design may have also been a culprit but in any case the introduction of 
Threat and Error Management (TEM) as a framework in scenario design and debriefing 
soon fixed the problem. The integration of CRM into simulation soon became more 
relevant. 


18.4. Extending CRM Training 


The irony about extending CRM training is that there is unanimity about the need to 
extend the training but in most cases the real message about the training (the 
management of human error) is not conveyed. The inadequate view is that CRM is 
about working together, being a team and communicating properly. As with the 
incipient variants of CRM, the same traps exist. A response comes from one airline 
which organised a complex seminar using pilots, cabin crew, station management and 
engineers as facilitators. The dominant theme of the seminar was to work together and 
“fuse” individual efforts to achieve operational goals. While the efforts to implement the 
training are admirable, strictly speaking it is not CRM training because it lacks the 
characteristics required of CRM: its error management focus, its need for application in 
simulation, and its continued reinforcement in check and training functions. The course 
is by all means an exercise in team building. The danger is that because of the use of line 
facilitators and the engagement of operations staff it is unfortunately associated with 
CRM training. 

Another airline put it succinctly: 


CRM training can be expanded to all operational areas of staff. If CRM is not expanded to 
other areas, training is not complete. That is why here in our region, many of us promote the 
“complete” or corporate training, but we have made mistakes in our attempts to expand it. 
We believed that complete training meant expanding training to all company areas, either 
operations or not; that CRM was the magic pill. Moreover, it was introduced in these 

workshops, but without clear conception and so it turned from operative training to cathartics 
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and evangelization of the environment used for different areas to exchange concerns about the 


other department. 
(Covello, 2009) 


18.5. Introduction of Threat and Error 
Management (TEM) 


The initial view about TEM from outside the USA was the trepidation that it was an 
invention to replace CRM. In some quarters, it was thought of as the newest generation 
of CRM. Fortunately, this view soon subsided and the appropriate context of TEM as 
a human factors and safety framework emerged. Additionally, LOSA became clearer 
as a data collection and analysis tool for monitoring normal operations. 

As TEM became more understood worldwide, it refreshed CRM and reshaped its 
original intent and error management context. There are pragmatic responses to the 
introduction of TEM: 


TEM makes CRM concepts easily understood especially when introduced early on because 
TEM is specific. It enables the “language challenge” which we sometimes face here in China 
to be better managed. 

(Mendoza, 2009) 


We also have a chance for the rebirth of operative training including TEM as a synthesis 
instruction, and LOFT as evidence of CRM theory. 
(Camelo, 2009) 


We can say that until TEM appeared we only focused on one aspect of the problem, just on 
human behaviour. With TEM, we can see the scenario where human performance takes 
place. It must be highlighted that TEM does not replace CRM training but gives us a more 
ample and complete approach. 

(Covello, 2009) 


18.6. The Future of CRM Outside the USA 


A significant revelation about responses on the future of CRM outside the USA revealed 
a preponderance of opinion towards local civil aviation authorities and its importance to 
the continued maintenance of CRM. Many of the responses while extolling the 
importance of the regulator also called for greater reinforcement of CRM education and 
qualification within the regulator’s jurisdiction: 
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Sad to note that CRM integration into our CAAP is not taken as a primary concern, 
probably the main reason is that none of those in the hierarchy even have a basic understanding 
of what CRM is all about and the leadership is politically appointed. 

(Mendoza, 2009) 


The lack of institutional resources to invest in CRM education within the regulator 
will be a perennial barrier towards airlines wanting to advance their programs. Devoting 
fiscal resources will suffer even more in the face of economic downturns where 
supplying basic social needs are more urgent. An avenue to explore in this regard is to 
gradually increase self-regulation by airlines (with the assistance of the local aviation 
safety interest groups) to improve their own CRM programs. 

Without constant influx of data such as LOSA and CRM training outcomes, the 
program loses its usability as a potent tool for managing harm to lives and assets. CRM 
therefore becomes static rather than a refreshing experience to learn more about 
avoiding mistakes. 


CRM as an operative tool might be denatured and can turn into a bureaucratic obligation to 
attend a course as part of the annual training. 
(Camelo, 2009) 


The role of the regulator in maintaining CRM cannot be de-emphasized, 
specifically with its responsibility for evaluating its local airline’s’ CRM faithfulness with 
both industry practice and ICAO recommendations. The task is daunting when in fact 
this chapter only contains conversational data among the airline population. General 
aviation data are not included and yet they too have relative responsibilities to implement 
and maintain CRM. Perhaps in closing, it is fitting to quote a universal experience by 
CRM developers not necessarily exclusive to the non-Anglo-Saxon world, where hope 
shapes the face of CRM: 


It is always an uphill climb when you take a proactive approach as it is difficult during 
presentation to management about initiatives to be done when one has no results to justify 
the ends. Nevertheless, in spite of all these difficulties, the rewards are invigorating as we get 
the rewards of educating the needful in this field. 
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Introduction 


Crew Resource Management (CRM) training was first introduced in the US military 
in the late 1980s (Prince and Salas, 1993). The impetus for CRM training in the US 
military came directly from commercial aviation. Based upon civil aviation models, early 
CRM courses were not well received by all military aviators. However, in the early 1990s 
the US Army, Navy and Air Force began funding CRM-related research, and great 
advances were made in terms of developing a research based model for delivering effective 
military CRM training (see Prince and Salas, 1993 for a review). Researchers and mili- 
tary aviators developed a CRM program that consisted of basic concepts, academics and 
skill sets relevant to the demands of military flight operations. In the US military, this 
model has remained largely unchanged for over a decade. Further, this research served 
as the basis for CRM training in many military services throughout the world. 

The purpose of this chapter is to build upon, and update, Prince and Salas’ (1993) 
review of CRM training in the military. The chapter will identify the unique 
challenges of military CRM training, provide examples of how CRM training is 
currently being used in the world’s militaries and review the evidence supporting the 
effectiveness of military CRM training. The chapter will close with a discussion of 
the future direction of CRM training in the military. 


19.1. Civilian Versus Military CRM Training 


A comparison of the mishap rates attributed to failures of CRM training in military 
and civilian aviation would appear to suggest that CRM training is less effective in the 
military. Table 19.1 provides a summary of six studies of civilian and military mishaps 
in which CRM failures were specifically categorized as causal factors. All of these 
studies used a common classification system called the human factors analysis and 
classification system (HFACS; see Wiegmann and Shappell, 2003 for more 
information). 

From Table 19.1, itis possible to conclude that CRM failures are identified as causal in 
aviation mishaps three times more often in the military as compared to civil aviation. This 
may lead to the conclusion that CRM training in commercial aviation is more effective 
than military aviation. However, this is a vast oversimplification of the issue. One 
explanation for the differences in the proportion of CRM-related mishaps may be the 
stark differences between the air carrier industry and military operations (see Prince and 
Salas, 1993 for a detailed discussion of these differences). There is arguably a much 
smaller margin for error in military as compared to civilian aviation. Military flight 
operations can be very complex as air crew utilize aircraft, weapons and sensor systems 
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Table 19.1 A comparison between the frequency with which CRM failures are identified as 
causal in aviation mishaps between military and civilian organizations 


Study Mishaps and sample CRM failure causal % 
Li and Harris (2006) 523 Chinese military 146 27.9 
Gibb and Olson (2008) 124 US Airforce 

Wiegmann and Shappell (1999) 226 US Navy 

Total military 873 military 

Shappell et al. (2007) 1,020 US commercial 











Gaur (2005) 48 Indian commercial 
Shappell and Wiegmann (2001) 164 US general aviation 


Total civilian 1,232 civilian 





while at the same time executing all the requirements that go with flying an aircraft. 
Due to factors such as the operational environment, mission timing constraints, multiple 
unit coordination and workload, it may be more difficult for military aviators to trap and 
mitigate errors than civilian aviators. It is also possible that due to the awareness of CRM 
skills and concepts that accrues from an effective CRM training program, perhaps military 
mishap investigators today are keen to list “breakdown in CRM,” or “CRM failure” to 
explain the mishap. Dekker (2005) noted that mishap investigators are often driven to find 
an underlying failure (or failures) that ultimately led up to and caused the mishap. 

In addition to the differences described by Prince and Salas (1993), there are also 
a number of CRM programmatic challenges in the military that may not exist in 
commercial organizations. For one, the primary concern of the military is mission 
accomplishment. There is arguably a different philosophical approach to CRM training in 
the military as compared to civilian applications. In most military settings, CRM 
training can be considered an operational training program. In the past, safety was not 
always a prioritized goal in military flight operations. To illustrate, the goal of US Naval 
CRM training is to “improve mission effectiveness by minimizing crew preventable 
errors, Maximizing crew coordination, and optimizing risk management” (Chief of 
Naval Operations, 2001). While safety is of great importance to military flight 
operations, it is not the purpose or primary aim of flight missions. 

Second, military personnel frequently change jobs. Military CRM trainers do not remain 
in their posts, or assignments, for very long. This may sometimes lead to shortfalls in 
continuity and “corporate memory” of the organization’s (squadron’s) CRM and safety 
training programs. Again, taking the US Navy as an example, naval aviators will 
generally only spend two to three years in a particular squadron before they move on to 
a different squadron or other non-flying assignment (e.g. staff job, higher education). 
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The US Air Force follows this personnel assignment practice as well. Further, even when 
at a squadron, the non-flying duties aviators perform also change regularly. In the US 
Navy it is likely that the squadron CRM facilitator may only perform that role for as 
little as a year before it is passed to another individual. The rationale for this practice is to 
provide officers with exposure to a broad base of squadron operations and leadership. 
The military is not a homogeneous organization, throughout military aviation, there are 
great variations in the type of aircraft (e.g. fighter, heavy, rotary wing and even tilt- 
rotor), missions performed (e.g. transportation, bombing, surveillance, special 
operations), operating environment (e.g. airfield, forward deployed, carrier based), 
number of crew (one to more than 20), and levels of automation (very little, to fully glass 
cockpit). As variable as the number and types of aircraft in the military aviation 
organization are the number and “type” of personnel flying these aircraft. In commercial 
aviation, most pilots are hired on to an airline after they have achieved significant 
aviation milestones. In military aviation, most “new hires” are college graduates with no 
previous flight experience. For these “nuggets,” experience and proficiency are gained 
on the job. That is not to say there is a dearth of experience in military aviation. On the 
contrary, there are a large number of senior pilots and air crew who garner substantial 
experience in their careers. This gives rise to the rank and experience gradient in 
military pilots and air crew. Rank is important in the military organization. In a given 
squadron, there are usually a fair number of lower ranking officers with lesser experience 
mixed with middle and senior ranking officers with greater levels of experience. Given 
the variety of missions, type of aircraft and personnel in military aviation, there is a need 
to balance the goal of having a common overarching program that is also tailored to the 
needs of the individual operators and specific aircraft. As was found with the early US 
Navy CRM training, a one-size fits all approach can become a one-size fits nobody. 
The military is a multidisciplinary organization, with many different cultures in 
terms of acceptance of CRM principles. Deal and Kennedy (2000) pointed out that most 
organizations possess a corporate culture. The same can be said of military organizations. 
Although military services each have their own traditions and “corporate” culture, 
within each service, individual units acquire their own “subcultures.” This stems in large 
part from their environment. Depending on the type of aircraft and mission, military 
aviation squadrons operate in different locations under different circumstances. In short, 
they have different operating environments that produce different requirements for 
success in that situation. This in turn gives rise to the culture or climate unique to a given 
squadron or aircraft community within military aviation. “The way we do things here” 
describes the effect of organizational culture on local practices within an organization 
(Bower, 1966). Sometimes this produces behaviors that are slow to change. This may 
partially explain why CRM was not widely embraced in its introduction to military 
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flight operations, and why even today, despite widespread integration and acceptance, 
there are varying degrees of qualitative implementation of CRM programs throughout 
military aviation. 


19.2. Current Military Approaches to CRM 
Training 


For all the research on military CRM in the past two decades, there is certainly no surfeit 
of literature or studies of military CRM programs. Further, what has been written is 
almost exclusively from a US military perspective. In an attempt to gather knowledge of 
the present state of CRM programs in military services worldwide, the authors 
contacted mid-grade officers, many of them in safety or CRM program billets, and 
queried them as to how their service is applying CRM training. Here the authors 
annotate a short summary of information obtained from their correspondence, 
interviews, in addition to published material. 


19.2.1. United States Navy and Marine Corps 


The US Navy and Marine Corps (collectively, the naval services) instituted CRM in the 
early 1990s (for a detailed discussion see O’Connor et al., in press). The program was 
developed on the basis of a substantial research program carried out at the Naval Air 
Warfare Center Training Systems Division (NAWCTSD). This led to the identification 
of seven CRM critical skills (decision-making, adaptability/flexibility, situational 
awareness, mission analysis, communication, assertiveness and leadership), and 

a theoretically grounded method for how to train these skills (for a more detailed 
discussion of the research used to develop the program, see Oser et al., 2001, 2000; 
Prince and Salas, 1993). 

The Navy’s CRM program is governed by a Chief of Naval Operations (CNO) 
Instruction— OPNAVINST 1542.7C. This instruction sets the basic administrative 
organization of the CRM program, and outlines a basic framework for CRM in the US 
Navy and Marine Corps. 

The architecture of the CRM program is a centrally controlled by the program 
model manager, but each aviation platform is given the latitude to administer its own 
CRM program tailored for its particular aircraft and mission. In a sense, the Navy 
and Marine Corps have 48 separate CRM programs (one for each type of aircraft the 
Navy and Marine Corps flies) each with a Curriculum Model Manager that ensures 
adherence to basic naval CRM academic principles, skills and program standardization. 
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The Curriculum Model Managers attend five days of training to prepare them for the 
role of both managing the program for their particular aircraft type, as well as training 
the CRM facilitators at each squadron. 

Although the last comprehensive update of the training curriculum was in 1999 by 
NAWCTSD (Oser et al., 2001), some Curriculum Program Managers have made 
exhaustive efforts to improve CRM in their communities. To illustrate, some of the 
commands that are responsible for training new aviators have drawn extensively on 
airline models for multi-place CRM training, while the MV-22 Osprey (a tilt rotor 
aircraft that can land and take off like a helicopter and fly like a fixed wing airplane) 
program has developed robust CRM courseware and models for their CRM program. 
The CRM curriculum in these communities greatly exceeds the basic minimum 
requirement in the CNO instruction. However, these exemplar programs are generally 
the exception. Most Curriculum Program Managers’ CRM programs reflect the basic 
academics required by the CNO instruction. Promulgating the latest academic themes 
in a “train the trainer” system is sometimes a challenge in a large organization. 

Although many high-risk civilian industries have embraced CRM (see Flin et al., 
2002) the same is not true of the non-aviation communities in the US Navy. Bridge 
Resource Management (BRM) was introduced into the curriculum of the Surface 
Warfare Officers School (the command that trains officers who will work on ships) in 
2006. However, it is still in the process of becoming properly established. The only other 
isolated examples of the use of CRM training by non-aviation personnel are naval 
medicine and navy diving (O’Connor and Muller, 2006). 


19.2.2. United States Air Force 


The Military Airlift Command (now Air Mobility Command) was the first military 
organization to formally embrace CRM training, implementing CRM instruction in 
1985 and co-sponsoring the pivotal Cockpit Resource Management Training 
Conference with NASA-Ames in 1987 (Orlady and Foushee, 1987). 

The Air Force Inspection and Safety Center developed a prototype course for single- 
seat fighters in 1990. In 1994, the Air Force formally mandated CRM training and 
assessment for all Air Force flight crew members. Air Force Instruction (AFI) 11-290 is 
the current document that formalizes the Air Forces CRM program (Secretary of the 
Air Force, 2001). This Instruction was formally adopted in 1998 and has remained 
essentially unchanged since then. It requires major commands to develop and manage 
tailored, mission-specific CRM training programs and mandates CRM training for 
all Air Force air crewmembers. The stated goals are to maximize operational 
effectiveness and preserve personnel and material resources. Six core CRM skill and 
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knowledge areas were specified: situational awareness; crew coordination/flight 
integrity; communication; risk management/decision-making; task management; and 
mission planning/debrief. 

As with the US Navy, while broad requirements are established in AFI 11-290, 
implementation is the responsibility of major commands, and these organizations have 
considerable flexibility regarding how these requirements will be met. As a result, the 
nature of CRM training can vary considerably across commands, and even across 
specific airframes within a command. For example, formal CRM instruction in Air 
Mobility Command during initial qualification training varies from 2.75 hours to 16 
hours depending upon aircraft type (Fisher, 2007). Most CRM instruction for air crews 
in operational commands is delivered by commercial trainers, either as part of 
a contracted integrated air crew training system or as stand-alone CRM instruction. All 
CRM training programs involve instructor-based training. Some also include 
computer-based training, workbooks or hands-on experience in training devices while 
others do not. 

CRM training developers and Air Force program managers often use existing data 
and experiences to help focus CRM training on areas of greatest need. For example, 
crews from several aircraft were observed during challenging simulator scenarios, and 
strengths and weaknesses were captured and then analyzed to identify CRM behaviors 
that are correlated with mission performance (e.g. Povenmire et al., 1989; Spiker et al., 
1999; Thompson et al., 1999; Wilson and Deen, 2003). These studies validated the 
importance of all six Air Force CRM skill areas in multi-crew operations. 

As formal CRM training courses were implemented across the Air Force, they 
seemed to be more readily embraced by aviators flying crew aircraft than by fighter and 
attack aircraft pilots (Karp et al., 1999). Nullmeyer et al. (2005) found that, while C-130 
Class A mishap reports tended to cite problems in all six CRM skill areas, human factors 
cited in F-16 and A-10 mishap reports were heavily skewed toward situation awareness 
and task management problems (see Figure 19.1). Coordination, flight integrity and 
mission planning problems are rarely mentioned in Air Force single-seat aircraft mishap 
reports. 

When AFI 11-290 was written in 1995, fourth generation CRM (as defined by 
Helmreich et al., 1999) represented the state-of-the-art in CRM instruction. Aspects of 
Helmreich’s fifth generation (error management) and sixth generation (threat and error 
management) CRM training have appeal, but the data now used by airlines to shape 
threat and error management training content are generally not available for Air Force 
CRM program managers or courseware developers. There is currently no requirement 
in AFI 11-290 to conduct a Line Oriented Safety Audit (LOSA) or analyze flight data 
recorder information, and the Air Force does not have a service-wide equivalent to the 
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Figure 19.1 Proportions of CRM-related factors in recent major US Air Force 
mishaps 





Proportions of Mishaps 








Aviation Safety Action Program (ASAP). These initiatives are being considered in 
several Air Force organizations, but the resources needed to implement such programs 
remain elusive. 


19.2.3. United States Army 


The US Army introduced its CRM training program, called Aircrew Coordination 
Training (ACT), in the early 1990s. The Army was the first US service to achieve 

a service-wide standardized CRM program. Since the inception of ACT, the US Army 
has invested significant resources to update its program, and in 2006 implemented 
ACT-E (Aircrew Coordination Training-Enhanced). 

In the US Army, all aviators are given initial ACT-E training during their primary 
flight training to become aviators. This training is broad, and covers the core academics 
and concepts of the ACT-E program. Once the aviator is assigned to an operational 
aviation squadron, he or she then receives annual ACT-E training in the form of 
a blended presentation by the squadron designated ACT-E instructor. 

The CRM skills of aviators are evaluated during every check flight (e.g. annual 
instrument check; annual aircraft proficiency check flight). Advanced ACT-E training is 
provided to the squadron ACT-E instructors by the US Army ACT-E program manager. 
This two-and-a-half day training teaches the ACT-E instructors how to access training 
media, deliver it to air crew for air crew recurrency ground training, and ensure its 
implementation in all check flights. 
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Central to the administration of the US Army ACT-E program is its program 
manager. The program manager maintains a great deal of standardization throughout 
Army aviation. Much of the program manager’s duties are outsourced to commercial 
vendors, but uniformed Army personnel occupy key positions in administration of the 
program, and generally have final say on programmatic and academic decisions. Under 
the program manager, Army ACT-E branches to six “subprograms,” one for each of the 
primary type of aircraft in Army aviation (heavy attack; reconnaissance; cargo; utility; 
fixed wing; and unmanned aerial systems). 

All squadrons send an aviator to receive the ACT-E instructor training. Once back in 
the field, the instructor retrieves the curriculum material (learning objectives, lesson 
plans and electronic media) from the program manager. ACT-E instructors are provided 
with the material for recurrent training by the program manager (unlike the US Air 
Force, or Navy, where the onus is not on the instructor to develop the materials). The 
training materials consist of electronic media, video case studies and slide presentations, 
to guide the aviators through a relevant discussion of ACT-E as it relates to the type of 
aircraft and mission. This ensures new case studies for a given aviator every time he or 
she participates in annual ACT-E training. 


19.2.4. United States Coast Guard 


As part of a program started in 2003, all pilots and air crew are required to participate in 
initial and annual Coast Guard CRM training (Commandant US Coastguard, 2005). 
The Coast Guard CRM training is very similar to that of the US Navy, addressing the 
same seven skills. However, as with the Air Force, the Coast Guard has also been looking 
to “best practices” from the commercial CRM industry, and recently started to 
incorporate them into their aviation CRM training. 

The Coast Guard also delivers CRM training to boat personnel, called Team 
Coordination Training (TCT). This was first introduced in 1991 and was revised on 
the basis of an analysis of mishaps in 1994 (see Hanson, 1996 for details). The most 
recent iteration of TCT is designed to increase team effectiveness and minimize human 
error in cutter, boat and command/control operations and activities (Commandant US 
Coastguard, 1998). The curriculum of the TCT course also consists of the same seven 
skills taught in the US Navy CRM program. The training is given by Coast Guard 
personnel, tailored towards personnel fulfilling particular roles within the Coast Guard, 
and varies in length and delivery method depending on who is receiving the training 
(see Commandant US Coastguard, 1998 for details). Hanson (1996) reported 
a reduction in minor navigational errors as a result of the first generation of TCT 
training. 
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19.2.5. Non-US Military 


Most of the world’s military services have a CRM program. As noted above, the authors 
contacted mid-grade officers, many of them in safety or CRM program billets, and 
queried them as to how their service is applying CRM training. In response to research 
requests, communications were received from: Royal Air Force (United Kingdom); 
Royal Australian Air Force; Royal Australian Navy; Italian Navy; Dutch Navy; Spanish 
Navy; South African Air Force; Taiwan Navy; and the Finnish Air Force. Most of the 
contacts here are officers who have attended US military training courses for CRM and 
safety programs. 

The influence of US military CRM programs, curricula and research can be seen in 
all the services above. In some cases, however, such as the UK, there is substantial 
independent research and subsequently a distinct CRM program is emerging. In the 
UK, study is under way to modify its CRM training. The aim is to shift the focus from 
error reduction and flight safety to human factors training that improves individual and 
team performance. In the Netherlands, the basic academic curriculum for military 
CRM programs is administered by the Netherlands Defense Academy (NLDA). 
Dedicated staff have imported much of the US Navy and Marine Corps CRM tenets, 
but have molded an implementation scheme that is highly integrated into the NLDA’s 
curriculum for its students. 

It was noted that all the services reviewed possess a basic CRM curriculum that is 
centered upon key academic topics and skill sets. Unsurprisingly, these topics and skills 
are very similar to those taught by the US military services. 

Most countries have a Ministry of Defense order or instruction that governs their 
CRM program. The Dutch military CRM program subscribes extensively to civilian 
CRM program governance promulgated by Dutch civil aviation authorities. In the case 
of the Australian military services, there is an Australian government Department of 
Defense instruction (DI-G OPS 40-4) that governs military CRM programs. Each 
military service has adopted this instruction as the service’’ CRM governing order. The 
result is continuity in CRM programs throughout the organization. 

There is considerable variability in CRM programs as regards recurrent training. 
Some services such as the Italian and South African CRM programs do not have 
prescribed recurrent requirements. In several services, recurrency consists merely of 
a ground training session; and in services that do conduct recurrent training, 
periodicity requirements range from annual to once every four years. In most 
services, CRM training is conducted by service members. In the South African Air 
Force (SAAF), efforts are under way to implement a CRM program taught by 
uniformed personnel. 
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For all the services above, CRM programs were implemented in the 1990s. Once 
the US military adopted CRM programs, it appears many non-US militaries followed 
suit. As was the case in the civilian world, the military is beginning to recognize the 
relevance of CRM training beyond aviation. In the SAAF there is an initiative to 
provide CRM to ground crews and air traffic controllers. The NLDA has integrated 
CRM training into the curriculum for its midshipmen. This is significant because not all 
midshipmen will go on to become aviators. The NLDA expects these young officers in 
the making to take these skills with them to their warfare areas wherever they may be— 
submarines, engineering offices, or surface ships. Another initiative in the Netherlands 
Navy is the implementation of Bridge Resource Management for officers on ships. 
Therefore, as CRM training is widely used by the militaries around the world, what 
evidence is there that they are getting a good return on the investment of time, money 
and personnel? 


19.3. The Effectiveness of Military CRM 
Training 


The Federal Aviation Authority (2004) states that for CRM training “it is vital that each 
training program be assessed to determine if CRM training is achieving its goals. Each 
organization should have a systematic assessment process. Assessment should track the 
effects of the training program so that critical topics for recurrent training may be 
identified and continuous improvements may be made in all other respects” (12: FAA, 
2004). This is also true for military CRM training. 

Table 19.2 lists 27 studies the authors have found in the literature reporting an 
evaluation of the effectiveness of CRM training in military organizations, and the type of 
evaluation that has been carried out (for a more detailed summary of these studies see: 
O’Connor et al., 2008, 2002; Salas et al., 2001, 2006). The evaluation methods were 
categorized in accordance with Kirkpatrick’s (1976) training evaluation hierarchy. 
Evaluating reactions are the equivalent to measuring satisfaction. For example, did the 
participants like the training? Learning is the second level in the hierarchy, and refers to 
“the principles, facts, and skills which were understood and absorbed by the participants” 
(Kirkpatrick, 1976, p. 11). Learning is made up of two components: attitudinal change 
and knowledge gains. Evaluation of behavioral changes is the assessment of whether 
knowledge learned in training actually transfers to behaviors on the job, or a similar 
simulated environment. The organizational level is the highest in Kirkpatrick’s (1976) 
hierarchy. The ultimate aim of any training program is to produce tangible evidence at an 
organizational level, such as an improvement in safety and productivity. 


Table 19.2 Evaluations of the effectiveness of military CRM programs (,/ indicates a positive finding, x a negative, or 
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US Navy 





Alkov (1991) 
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41 helicopter 
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27 helicopter pilots 
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It can be seen from Table 19.2 that, on the whole, military CRM training seems to 
be having a positive effect at each of Kirkpatrick’s (1976) levels of training evaluation. 
However, a number of caveats need to be made regarding the evaluation data: 


1. The evaluations lack currency. Almost 90% of these studies were carried out 
more than a decade ago, with none carried out in the last five years. Therefore, 
there is little to guide thinking on the current state of military CRM 
training beyond anecdotal evidence. 


2. The evaluations are heavily US centric. Apart from four studies, the evaluations 
were exclusively carried out within the US military. 


3. The evaluations are generally of specific populations of a small number of aviators. 
The majority of the studies report evaluations of small numbers of participants 
from a specific aviation community, who all attended the same training 
course (e.g. Salas et al., 1999; Stout et al., 1996). Thus, the findings may not 
be generalizable to the military organization as a whole, and may partially be 
a reflection of the tutors’ teaching ability, rather than the actual content of 
the training. 


Therefore, although these findings are encouraging, there is a need for military 
organizations to conduct large-scale evaluations of the current effectiveness of their 
CRM training program. Without this information it is impossible to assess the 
effectiveness of the program, and identify whether the organization is obtaining an 
acceptable return on investment. Further, just because a program was successful a decade 
ago, does not mean this continues to be the case. There is a great deal of information 
available on how CRM training can be evaluated (e.g. Flin et al., 2008). It is suggested 
that the world’s militaries should use this information to evaluate the effectiveness at as 
many levels of Kirkpatrick’s evaluation hierarchy as is feasible. 


19.4. The Future of CRM Training in the 
Military 


The authors have no doubt that military aviators will continue to receive CRM training 
for a long time to come. However, the training will have to adapt to address the 
automation issues associated with the new advanced aircraft (e.g. Joint Strike Fighter, 
Euro Fighter, F-22 Raptor and V-22 Osprey). CRM in highly automated aircraft 
presents special challenges for aviators. Just as has been the case with CRM training in 
commercial aviation, there is a need to address the change in crew coordination 
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dynamics that can result from a more automated aircraft. The other big area that is 
suggested that CRM should be applied is to non-aviation military communities. 
Given the almost two decades that have passed since it was first used in military 
aviation, the dearth of non-aviation applications in the world’s militaries is surprising. In 
high-risk civilian industries, those that adopted it first were, unsurprisingly, involved in 
the aviation business (e.g. aviation maintenance, air traffic control). However, CRM 
training has also begun to be used in a number of other high-reliability industries 
unrelated to aviation such as: nuclear power generation, anesthetics, the maritime 
industry and offshore oil and gas production (see Flin et al., 2002 for a review). It is 
suggested that given the experience with delivery CRM training in aviation, and the 
range of other high-risk military occupation, the military is well placed to introduce 
CRM into other domains. One particular domain in which CRM training should be 
more widely used is for the training of Unmanned Aerial Systems (UASs) operators. 
UASs have become an indispensible asset in military operations. A seemingly 
insatiable demand for the capabilities that UASs bring to the battlefield have all the US 
services struggling to train enough operators to keep pace with rapidly growing demand. 
A number of new human factors issues accompany this major move toward remotely 
piloted aerial vehicles including many pertaining to resource management. McCarley 
and Wickens (2005) identified some of the more salient emerging human factors 
challenges including the loss of sensory cues such as ambient visual information, 
kinesthetic/vestibular inputs and sound. Second, UASs differ dramatically in the degree 
to which flight control is automated. An emerging challenge is to develop and optimize 
procedures for responding to automation failures, including common expectations 
among UAS operators and air traffic controllers. A third set of issues pertains to the 
selection, composition and training of UAS flight crews. The steepness of demand for 
UAS operators is forcing the services to think beyond the traditional approach of 
assigning experienced pilots to the task of controlling UASs. All the US services are 
either using non-pilots as UAS controller trainees or are actively considering that option. 
In the past, mishap rates were much higher for UASs than for manned military 
aviation. To illustrate, since it’s inception the cumulative mishap rate for the Pioneer 
UAS is 334 per 100,000 flight hours compared with a rate of about 2 per 100,000 in US 
naval aviation in recent years (Thompson et al., 2005). Williams (2004) analyzed US 
Army, Navy and Air Force UAS mishaps. The percent of human factors mishaps varied 
across UASs from 21% to 68%. Those that involved causal human factors tended to 
involve issues with alerts/alarms, display design, procedural errors or skill-based errors 
(Tvaryanas et al., 2005) analyzed a ten-year cross-section of 221 US Army, Navy and Air 
Force UAS mishaps. Overall 60% of mishaps were human related, ranging from 32% to 
69% depending on the platform. Further, specific human factors problems varied 
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considerably across systems. Air Force factors tended to involve the operator station and 
channelized attention. Army factors included overconfidence, crew coordination, 
communication and organizational factors related to procedural guidance, publications 
and training programs. Navy issues involved organizational and supervision issues, 
operator station design and crew factors of channelized attention, oversaturation, 
distraction, inattention and complacency. 

These reviews reveal at least three important aspects of UAS operations that will 
impact CRM training. First, human factors are commonly cited in mishap reports 
indicating that there is ample opportunity to improve safety. Second, problems appear to 
vary considerably across platforms and organizations. This suggests that Helmreich and 
colleagues’ threat and error management approach to CRM training content would 
have merit (see Helmreich et al., 2001 for more details). Third, the traditional Army, 
Navy, or Air Force CRM content areas are well represented in the human factors that 
are emerging as the primary problem areas. However, it is crucial that the training has 
been tailored to the needs of UAS operators. 

As has been found in other CRM applications, simply replacing “aviator” with 
“UAS operator” and giving the standard CRM training is not likely to be successful. If 
CRM is to be adapted for use in other domains, the training materials must be 
customized. For the training to be effective it is imperative that the skills that are 
required have been identified through a training needs analysis, the psychological 
concepts are translated into the language of the participants receiving the training, and 
that relevant practical examples and case studies are used to illustrate the concepts. The 
training is likely to be ineffective unless examples poignant to the particular domain are 
used. One of the main criticisms of participants of the early civilian aviation CRM 
courses was that there was too much psychological theory and not enough relevance to 
aviation. “I am not suggesting the mindless import of existing programs; rather, aviation 
experience should be used as a template for developing data driven actions reflecting the 
unique situation of each organization” (Helmreich, 2000, p. 784). 


19.5. Conclusion 


CRM training has become much more widely used in the world’s militaries since Prince 
and Salas’ (1993) review. However, as with military hardware, a training program must 
not remain stagnant or it is in danger of becoming obsolete and ineffective. It is 
suggested that the world’s militaries should revisit their CRM training programs, and 
reassess where the training continues to be effective and where it needs to be changed. 
This will take time and money. However, when one considers both the price of 

a modern military aircraft (more than $50 million for an F-18 Hornet E/F; US Navy, 
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2006) and the cost of training someone to fly it ($1.5 million for basic jet flight training; 
About: US Military, 2007), an effective CRM training program is a cost-effective 
method for improving for both safety and mission performance. 
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Introduction 


The first edition version of this chapter, published in 1993, and written by my late father, 
Harry Orlady, stated “that pilots already have the technical skill and knowledge tradi- 
tionally required of an airline pilot” (Orlady, 1990a). It was assumed that by the time 
pilots began flying in airline cockpits they would have a wealth of knowledge and 
experience from which to draw. Moreover, it was assumed that a pilot would have a fair 
period of time flying as a co-pilot or first officer before he/she accepted the command 
position of captain. 

The aviation industry has always been subject to economic cycles of supply and 
demand. There was a time in the early 1970s, for example, when the pilot supply was 
lean, in part due to so many military pilots occupied with wartime activities. Some major 
airlines hired pilots with very little flight time. For many of these pilots, 15 years passed 
before they became captains. Virtually all flew as flight engineers, some with the dubious 
distinction of flying as a B737 flight engineer—dubious as the 737 had been designed to 
fly with two pilots, a decision met with great resistance by some pilot labor groups. 

What knowledge, skills and experience are required for today’s airline pilots? Are 
minimum standards sufficient? What are the challenges for training? What does crew 
resource management help to address or to mitigate? What, if anything, does it hinder or 
allow to lie latent? This chapter will attempt to examine some of the current issues in 
pilot training and challenges for the future. It is not intended to be a comprehensive 
review of all airline training programs or training regulations but rather to focus on 
selected areas that have gained attention from recent events in the aviation industry. 
Some discussion will center on recent events occurring in the USA. However, very 
similar issues have been raised and are being discussed in other parts of the world. 


20.1. Training—The Primary Interface 


It is difficult to underscore the importance of training. Many have stated that the most 
important safety feature on any airplane is a well-trained, highly motivated and 
professional pilot (Prater, 2009; Hogeman, 2009; Sullenberger, 2009). The training 
process furnishes the primary interface between manufacturers, the airline operators, the 
environment in which they must be operated and the pilots that fly them. The 
environment in which these airplanes must be operated has become more complex. The 
airplanes have become both more sophisticated and more expensive. These critical 
interfaces between the manufacturers, the airline companies, the pilots and the 
environment, and the pilot training associated with them have become even more 
important. 
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Unfortunately, difficult economic conditions, financial strains, questionable 
management and unstable market conditions have also become a prominent variable in 
this interface. Some major airlines have endured bankruptcies, aircraft fleet reductions 
and accompanying pilot furloughs. The operating environment for the regional carriers 
similarly can be quite dynamic as they respond both to market conditions and to varying 
code-sharing agreements with their larger, network carriers. It is not unusual for some 
regional carriers to open and close pilot domicile bases as they accept new routes and 
close others, sometimes on a seasonal basis. 

Airline pilot training is expensive and represents a significant cost center to the 
airline. Some pilots are beginning their flying careers with debt in excess of $100,000. 
Working rules have become more intricate, airplanes and training aids have become 
even more highly developed, and the airplanes must be operated in an increasingly 
complex aviation system as the new technology of Next Gen approaches. Acceptance 
of full utilization of simulation by the operators, the pilots and regulatory 
organizations (e.g. the Federal Aviation Administration) has alleviated some of that 
expense, but it has by no means eliminated the real-world economics involved in pilot 
training. 


20.2. The “Minimum Standard” and Recent 
Events 


“There is no training gap; we have the exact same standards as the big carriers.” 
Statement from Roger Cohen, president of the Regional Airline Association, as quoted 
in the Seattle Times, May 17, 2009. 

Regulators—in the USA, the FAA—specify how training is to be conducted and 
evaluated. These regulations specify a minimum standard to be met. However, many 
carriers voluntarily exceed this standard based on an assessment of their training needs 
and frequently supported by information received from voluntary reporting programs 
such as ASAP and FOQA (Aviation Safety Action Program and Flight Operations 
Quality Assurance). The collection, protection and feedback of this information 
into training programs and scenario design has been one of the major improvements 
to training in the last ten years, although it has not always been without some 
challenges. 

In January 2009, the FAA proposed major changes to update training standards after 
several years of collaborative work and discussions with industry and labor association 
representatives (NPRM FAA-2008-0677). At this point in the regulatory process, the 
comment period for the Notice to Proposed Rulemaking (NPRM) has closed, and 
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review of comments will begin. Ironically, some recent accidents in the USA have 
greatly increased interest in airline pilot training, minimum standards and Crew 
Resource Management (CRM). 

On January 15, 2009, US Airways flight 1549 departed New York’s LaGuardia airport 
and suffered a loss of thrust in both engines as a result of bird strikes. Captain Chesley 
Sullenberger and his crew adroitly landed on the Hudson River and successfully evacuated 
150 passengers from their Airbus A-320. Captain Sullenberger has been effusive in 
praising the contributions of his first officer, three flight attendants, two jump-seaters, the 
air traffic controller and of first responders to the accident scene. The performance of the 
crew was exemplary; CRM at its finest. The NTSB held their public hearing on the 
Hudson “landing,” as they phrased it, in June 2009. However, months before Sullenberger 
testified before keenly interested members of the US House of Representative’s 
Committee on Transportation and Infrastructure on February 24, 2009 that: 


I am worried that the airline piloting profession will not be able to continue to attract the best 
and the brightest. The current experience and skills of our country’s professional airline pilots 
come from investments made years ago when we were able to attract the ambitious, talented 
people who now frequently seek lucrative professional careers. That past investment was an 
indispensible element in our commercial aviation infrastructure... We must not let the 
economic and financial pressures detract from a focus on constantly improving our safety 
measures and engaging in ongoing and comprehensive training. In aviation, the bottom line is 
that the single most important piece of safety equipment is an experienced, well-trained pilot. 
(Sullenberger, 2009) 


On February 12, 2009, Colgan Air flight 3407, a Continental express carrier 
Bombardier Dash 8 Q400, crashed a mile northeast of the outer marker while on 
approach to Buffalo Niagara International Airport, killing the 49 people onboard the 
aircraft and one individual on the ground. The National Transportation Safety Board 
held its public hearing during May 12-14, 2009. Their final report has not been released 
so speculation as to probable cause would be both inappropriate and premature. 
However, concerns were raised to a significant level during the hearing in numerous 
areas including pilot training, managerial responsibility and pilot professionalism. 

Partially as a response to this attention, the new FAA Administrator, the Honorable 
Randy Babbitt, quickly assembled major and regional airline officials, labor association 
representatives and FAA officials at a “Call to Action” meeting (6/15/09) in 
Washington, DC. The purpose of the meeting was to generate an agenda for 15 
additional meetings that would be held around the USA with the stated purpose of 
participants jointly identifying best practices and key initiatives regarding pilot training, 
cockpit discipline and other areas that could be voluntarily incorporated by operators. 
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Stemming in part from the Colgan accident and other recent regional airline crashes, 
both legislative bodies of the US government, the House of Representatives and the 
Senate, convened hearings on aviation safety and pilot training. Legislative actions 
resulted from these hearings, and on July 29, 2009, the Committee on Transportation 
and Infrastructure and the Subcommittee on Aviation introduced House Resolution 
(HR) 3371, the Airline Safety and Pilot Training Improvement Act of 2009. 

As the Subcommittee Chairman, Jerry FE Costello (Illinois), stated, 


Our bill is a comprehensive effort to consolidate what we know industry-wide about aviation 
safety to improve safety performance going forward. The more we looked at these issues, the 
more it became apparent that information about pilot training and safety programs is not 
readily available. There is a lot of information regarding industry best practices that is not 
being shared. This bill not only increased trained standards, but it seeks to identify what 
airlines are doing right in regard to safety and make this information a resource for others. 


The press release for the bill includes the following points: 


m Requires the FAA to ensure that pilots are trained on stall recovery, upset recover 
and that airlines provide remedial training. 


m Establishes comprehensive pre-employment screening of prospective pilots 
including an assessment of a pilot’s skills, aptitudes, airmanship and suitability 
for functioning in the airline’s operational environment. 


m Requires all airline pilots to hold an FAA Airline Transport License (1,500 
minimum flight hours required). 


m Requires airlines to establish pilot mentoring programs, create Pilot Professional 
Development Committees, modify training to accommodate new-hire pilots 
with different levels and types of flight experience and provide leadership and 
command training to pilots in command. 


m Directs the FAA to update and implement a new pilot flight and duty time rule and 
fatigue risk management plans to more adequately track scientific research in the 
field of fatigue. It also requires air carriers to create fatigue risk management 
systems approved by the FAA. (Berard, 2009) 


As another data point in this timeline of notable events affecting pilot training, FAA 
Administrator Babbitt responded quickly to the last bulleted item, chartering an aviation 
rulemaking committee (ARC) to address updating pilot flight and duty time rules with 
recommendations due to the FAA by September 1, 2009, an extraordinarily short 
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45-day timeframe. As a point of comparison, the safety management systems aviation 
rulemaking committee (the SMS ARC) was chartered in February 2009 and will 
provide rulemaking suggestions to the FAA in a process expected to last three years. 


20.3. Training Presents a Corporate 
Opportunity 


There are some major differences in how training is conducted at different airlines, 
despite a stated goal to have “one level of safety.” Interestingly enough, regional airlines 
were not required to adhere to the same standards as the larger airlines (FAR Part 121) 
until the mid 1990s. Depending on your perspective, “one level of safety” was adopted 
or imposed by the FAA in 1995 and had it roots from efforts of the Air Line Pilots 
Association as they sought to improve the operating conditions of their members flying 
for cargo carriers. 

Training provides a wonderful window of opportunity for an airline. It is the 
opportunity to expose and indoctrinate their employees with what Degani and Wiener 
eloquently described in 1994 as the “Four Ps” (Degani and Wiener, 1994). The airline’s 
corporate philosophy, policies, procedures and practices are key ingredients in any 
operation. They are the ingredients that constitute the organizational culture or as 
Professor Richard Hackman referred to it—the organizational shell (Hackman, 1986). 
An organizational culture with attributes that are consistent, clear, generative, 
accountable and “just” should result in a robust safety culture at the airline. Put more 
simply, training should represent “how we fly our airplanes” at a particular carrier. 

One challenge with having training embody a corporation’s “Four Ps” is that 
frequently the training is not conducted in-house or by the carrier itself. Sometimes, 
another company’s simulators are used—which may or may not have the same cockpit 
setup regarding the location of instruments and switches that the pilots will see on the 
line in their company’s aircraft. Sometimes, contract instructors are used who may not 
have a great deal of familiarity with a company’s specific procedures. None of this is to 
suggest that there are not a number of high-quality training organizations, but rather to 
suggest that this training must be coordinated. 


20.4. Crew Resource Management 


There have been major changes in pilot training. CRM is neither a new nor an 
unfamiliar topic. As Harvard Professor Richard Hackman stated, “...the overall task of 
successfully completing a flying mission is always a team task” (Hackman, 1986). 
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Virtually all airline pilot training is conducted in a crew context. Evaluation and 
checking also include crew performance assessments. These assessments evaluate crew 
performance on traditional parameters but also include an assessment on behavioral 
aspects such as leadership, team building, conflict resolution and situational awareness. 
Professor Robert Helmreich and his associates working on the NASA/University of 
Texas project pioneered both the research and implementation efforts in this area, for 
example in the development of behavioral and crew performance markers and 
implementation tools such as the Line/LOS checklist. 

Acknowledgment of the importance of the crew concept and, in particular, the 
captain setting a constructive tone in the cockpit was not limited to the USA. As was 
succinctly stated by a representative of the New Zealand Air Line Pilots Association 
(NZALPA) during a Royal Commission inquiry into the 1979 crash of Air New 
Zealand 901 into Mt Erebus, Antarctica, “Indeed, it would not be going too far to say 
that, if a pilot in command were to create an atmosphere whereby one of his crew 
members would be hesitant to comment on any action then he would be failing in his 
duty as pilot in command” (Vette and Macdonald, 1983). 

CRM enhanced the role of the co-pilot or first officer while reinforcing that there is 
only one captain in the cockpit and not two co-captains. Properly implemented, CRM 
should enhance a captain’s authority rather than diminish it. CRM has also emphasized 
the importance of the crewmember serving as the pilot not flying (PNF). In fact, some 
aircraft manufacturers as well as the already cited proposed changes to Federal Aviation 
Regulations (FARs) regarding training, refer to crewmembers as the “pilot flying” (PF) 
and have replaced the term “pilot not flying” (PNF) with “pilot monitoring” (PM). 
This substitution is intended to reinforce further the responsibilities of the non-flying 
pilot who is generally handling ATC communications and checklists, among many other 
responsibilities. While this wording change may enhance the perception of contributory 
value of this position, the wording is not the preferred choice of this author. Quite 
simply, in today’s advanced technology, both pilots have significant monitoring 
responsibilities; it is how today’s modern aircraft are “flown.” Especially with advanced 
technology airplanes with “glass-cockpits,’ both pilots have strong monitoring 
responsibilities. One would not want to infer that the “pilot flying” does not have any 
monitoring responsibilities. Furthermore, manufacturers and operators are discovering 
that we have a lot more to learn about effective monitoring. 

All CRM training programs will also stress resource management—the importance 
of coordination and effective utilization of other resources outside the cockpit—for 
example, flight attendants, dispatch and maintenance personnel. Many airlines schedule 
their recurrent training jointly with pilots and flight attendants. This training can be very 
helpful and enlightening. It frequently highlights different perspectives, communication 
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styles and workload levels of the two groups. On occasion, it leads to a constructive 
discovery of discrepancies with company-supplied manuals as to basic procedures or 
responses to an irregular or emergency operation. 


20.5. Typical Types of Airline Pilot Training 
Courses 


Traditionally, airline pilot training was concerned with developing and maintaining 
technical skills. While this focus remains, clearly there is a behavioral aspect that is 
represented in almost all training, again lending support to the crew concept and support 
for CRM. There are four basic types of formal training or courses. Each assumes 

a reasonably well-qualified pilot who has the skills and knowledge that are appropriate 
for his or her position. The training may be conducted in-house by the airline or using 
contract vendors—sometimes with their own instructors. 

The first training usually provided by an airline is for new hires. For some pilots, it 
includes an upgrading of skills and knowledge, but its main purpose is simply to teach 
the aviator what he or she needs to know to fly the company’s airplanes on the 
company’s routes. It frequently contains a general introduction and indoctrination into 
philosophy and procedures of the particular airline. For pilots with military 
backgrounds, these courses also serve to bridge the gap as they begin flying in the 
civilian sector. The second type of training has been called upgrade training. Here the 
purpose is to prepare the pilot for a change in status, such as from co-pilot to captain. A 
third type of training is transition or conversion training on a specific aircraft, for 
example a pilot transitioning within his airline from flying as a co-pilot on a B737 to 
flying as a co-pilot on a B767. The objective of transition training is to teach pilots how 
to fly this different airplane safely and efficiently in the company’s line operations. 

A fourth type of traditional airline pilot training is recurrent training. It affects 
all airline pilots. Recurrent training in the USA is mandated by FARs at specified 
times—traditionally every six months for captains and annually for first officers. The 
purpose of recurrent training is to ensure that the pilot has maintained his or her 
proficiency and the skills and knowledge required to fly as an airline pilot on a specific 
range of equipment. A secondary purpose is to reinforce knowledge of the latest 
operational information affecting the specific equipment involved. 

The time interval required for this training has changed in the last 10-15 years and 
can vary with different carriers. In the USA, the intervening time between sessions of 
recurrent training has lengthened for most carriers. The most common modification 
substitutes one three- or four-day session of training annually for the more usual two 
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periods of two days with each period six months apart. A condition for FAA approval of 
this modification for some carriers was the assurance that the airline would place 
considerable emphasis on CRM during each annual training period. And indeed, 

a number of carriers include flight attendants during joint groundschool training with 
significant positive feedback received from both groups regarding the training 
experience. Some carriers with an approved Advanced Qualification Program (AQP) 
can extend their recurrent training interval to nine months. In this scenario, a pilot 
would alternate receiving a checkride and recurrent training session every nine months. 
The intent and hope of these programs is that the extended interval for training can save 
costs while not adversely affecting safety. 

In many countries there is no ready source of previously trained potential airline 
pilots. Therefore, airlines have had to start ab initio training on their own. Dedicated, 
airline-specific ab initio training for prospective airline pilots started in Europe in the 
1950s (Johnston, 1991). It is a fifth type of training and remains a relatively new concept 
for many US and Third World airlines. When economic conditions force a downturn in 
the industry with accompanying pilot furloughs, frequently the supply of pilots 
outweighs the demand. Increasing the mandatory retirement age for pilots to 65 in the 
USA has also added to the supply. Ab initio training has received limited support among 
US carriers, as a result, at least at this time. 


20.6. Selection of Training Devices 


The high fidelity of simulator training devices used today is quite extraordinary. They 
allow us to have very realistic scenarios. But not all training requires a full-motion 
simulator. And the simulators should not be modified to the point where they do not 
respond as the actual aircraft would respond. This point was raised during the NTSB 
hearing for American Airlines flight 580 as it was revealed that some simulator 
parameters had been modified to further amplify some aircraft handling characteristics. 
The intentions by American clearly were honorable. However, a negative transfer of 
training will occur if the devices do not represent what will be experienced in the 
cockpit during line operations. The NTSB’s complete report, NTSB AAR-04/04, is 
available on their website (www.ntsb.gov). 

One of the lessons learned with the advent of glass cockpits is that part-task trainers 
can be very effective for training automation. Some of these devices are better than 
others. For example, a single-path device might give the mistaken impression to the pilot 
that there is only one way to accomplish a task. Some of the first FMC training devices 
were limited in this regard. It was frustrating as it was clearly obvious to the pilot that 
there were multiple paths to accomplish the task. 
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Computer-based training (CBT) is used by many carriers to cover topics such as 
security issues, carriage of hazardous cargo materials, winter operations, etc. Some 
carriers also have developed sophisticated computer-based training for their aircraft 
groundschool. At some large network carriers, pilots cover the entire aircraft 
groundschool using the CBT. Effective training will combine CBT with instructor 
interface, whether it is with classroom review sessions or with the use of a part-task 
trainer or simulator to review and reinforce the computer-based training. 

The quality of computer-based training can vary quite a bit. The important point to 
note is the value of reinforcing the training with a qualified instructor. Some carriers 
ensure that the training is tailored to their particular operation, to their checklists, to 
their procedures—that it supports the company’s “Four Ps,” as mentioned earlier. It is 
one of the more important points of training—and one, unfortunately, that is often 
missed by training designers. The training must represent how the aircraft are flown on 
the line. Otherwise, it risks being quickly dismissed by pilots who may conclude that it is 
irrelevant since it is not consistent with actual line operations. 


20.7. Training Automation and How Best to 
Monitor Performance of Automated Systems 


Over the years, we have learned a great deal about training automation. Initially, many 
airlines developed their own philosophy of automation and an introduction to 
automation course for those making their initial transition to an advanced technology 
aircraft. In its “National Plan to Enhance Aviation Safety through Human Factors” the 
Air Transport Association Human Factors Task Force summarized the following in 1989: 


During the 1970s and early 1980s, the concept of automating as much as possible was 
considered appropriate. The expected benefits were a reduction in pilot workload and 
increased safety... Although many of these benefits have been realized, serious questions 
have arisen and incidents /accidents have occurred which question the underlying assumption 
that the maximum available automation is always appropriate, or that we understand how to 
design automated systems so that they are fully compatible with the capabilities and 
limitations of the humans in the system. 

(ATA, 1989) 


Automation surprises happen to most pilots and have been studied by many 
researchers. A senior Aer Lingus captain shared a story with this author about approaching 
New York’s John F Kennedy Airport, arriving from Europe. The crew was forced to abort 
their approach and execute a go-around due to landing traffic that had failed to clear the 
runway. As they began their climb to go around, the power levels were not pushed forward 
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sufficiently to be seated in what, for this aircraft, was the takeoft/go-around detent 
position. As they climbed, the pilots raised the landing gear. However, this action 
conflicted with the designed logic of the automation. Since the power levers had not 
“clicked,” the aircraft’s warning systems believed that the aircraft was still attempting to 
land and was obviously not in landing configuration as the landing gear now was not 
“down and locked.” Therefore, during this critical flight period and while receiving 
special go-around instructions from JFK Tower, the aural alerting system began repeatedly 
announcing very loudly, “TOO LOW, GEAR, TOO LOW, GEAR.” The statement of 
the captain describing the incident was classic, “While I was quite confident that we were 
notin imminent danger of crashing the aircraft, I really did not know what was going on.” 

Frequently, the logic of surprising or unintended automation events, whether caused 
by the operators or from system design, is not fully understood until after the event. 
Even the best-designed systems can provide surprises that had not been conceived by the 
manufacturer. Training and procedures must prepare crews to remember their priorities, 
to use checklists as appropriate, but not to abandon basic airmanship or, frankly, their 
own cognitive ability to handle situations. 


20.8. Cockpit Workload 


Cockpit workload is an important and elusive concept (Wiener, 1985; Wiener and 
Curry, 1980). What is often forgotten is the importance of training in evaluating the 
workload that is imposed on operating pilots. A task that is very difficult for a poorly 
trained person can be a simple task for one adequately trained (Orlady, 1991). 

Pilots need to understand workload and how it affects their performance. We learned 
a great deal about workload and crew coordination with the introduction of advanced 
technology aircraft. It was (and is) very easy to become preoccupied, for example, with 
programming the flight management computer (FMC). It is important to have standard 
operating procedures that clarify programming responsibilities. It is very easy for an 
incompetent PNF (PM) to really mess things up and, consequently, to significantly 
increase the workload for the PE CRM has an important role here in terms of 
communicating workload overloads and with setting task prioritization. 


20.9. A Negative Consequence of CRM 
or Misapplication of CRM Principles? 


As a former CRM manager for a major US airline, it has been interesting to watch some 
of the training ramifications as we have embraced CRM over the last 25 years. Pilots are 
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evaluated on their crew performance and topics such as leadership, followership, conflict 
resolution and threat and error management. At many airlines, the entire crew can fail 
a checkride, as opposed to a single crewmember. It is also possible to separate a crew in 
order to identify one crewmember whose performance is not satisfactory. This situation, 
while difficult, can require some level of sophistication and training on the part of the 
organization, instructor and examiner. 

Training must develop crew skills but also expose weaknesses and over-reliance by 
weaker crewmembers. One of the fundamental tenets of Advanced Qualification 
Programs (FAA Advisory Circular 120-54) is that pilots are “trained to proficiency.’ If 
standards are not met in a particular area, the pilot will be trained until proficient and 
then checked. One of the potential dilemmas of this scenario is whether or not the 
narrowness of the recheck fails to catch a larger deficiency. 


20.10. Shortened Training Courses 


As a general statement, the length of a training course, sometimes referred to as the 
training footprint, has shrunk. For example, at one US carrier, the original Airbus 
320 course lasted eight weeks. It has since shrunk to six weeks and now to four weeks, 
which matches the same reduced training footprint as other aircraft at this airline. 

Training managers will advocate that reduction of training times is possible because 
the training is superior, more tightly designed to make more use of the time available, 
and that we have more sophisticated training devices—whether or not a full-motion 
simulator, computer-based training, or part-task trainer as might be used to train flight 
management systems. There is some truth to all of those statements. Is there anything 
negative, however? 

Captain Rory Kay, Executive Air Safety chair for the Air Line Pilots Association, 
made the following observation during his closing remarks at ALPA’s 55th Air Safety 
Forum: 


Few airlines can claim to be making money these days, and when it comes to looking for cost 
cutting, pilot training has very clearly become a soft target because there is no instant 
perceptible effect from reducing it, just an unquantifiable increase in risk, which perhaps with 
this current string of automation failures and widely varying crew responses, is becoming more 
quantifiable now. Can anyone here tell me that their training footprints have increased? I 
have been with my airline for 20 years, and I have watched a slow but continuous reduction 
in training footprints, a pattern that is common throughout the industry. I think it is high 
time for the regulators and airline managers to accept that we are starting to see costly 
incidents and tragic accidents occur that can be traced back to inadequate training and 


Airline Pilot Training Today and Tomorrow + Chapter 20 481 


insufficient practice of basic skills. The claims that we “train to proficiency” should now be 
challenged and opened to debate as to its real meaning. 
(Kay, 2009) 


20.11. Individual Needs 


There will be a greater effort to identify and meet individual needs as part of the move 
toward the “training to proficiency” or “train to competency” concepts, as discussed in 
Advanced Qualification Programs (AQP). While these needs may or may not be job 
related, they do affect pilot behavior and the efficiency of the training process. 

Individual needs can vary depending on recent and total experience, motivation, 
airline philosophy (particularly in cases of airline sales or mergers), individual capability, 
an elusive characteristic called “computer literacy” and many other things. Effective 
training to meet these sometimes different needs requires savvy instructors able to tailor 
their instruction. 


20.12. Individuals who Fail to Perform 


There are at least two types of competency issues. One refers to the so-called 
“recalcitrant trainee,” or the trainee who does not seem to fit in with or comprehend the 
“team approach,” or who seems to have an over-reliance on it. Such pilots have been 
called the “wrong stuff” (Helmreich and Wilhelm, 1989). While this may not always be 
an apt metaphor, it seems inevitable there will be a small number of individuals who do 
not have and cannot or will not acquire the additional skills that have become essential 
(Helmreich et al., 1989). This can be an extremely sensitive area and one with some 
liability and other legal concerns. 

Another related issue is the individual who puts forth the effort, has a good attitude, 
but simply is not performing to an acceptable standard. Frequently, these individuals are 
given additional training and counseling, and are assigned to special instructors. While 
the reasons for subpar performance should be examined and addressed, at some point, if 
an individual is not meeting performance standards, it only prolongs and exacerbates 
problems to continue training. Even ALPA’s code of ethics attempts to address this 
extremely sensitive issue, “An air line pilot... will not falsely or maliciously injure the 
professional reputation, prospects, or job security of another pilot, yet if he knows of 
professional incompetence, he will not shrink from revealing this to the proper 
authorities within ALPA, so that the weak member may be brought up to the standards 
demanded, or ALPA and profession alike may be rid of one unworthy to share its 
rewards” (ALPA, 1956). This is a difficult concept accompanied by harsh-sounding 
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words, but also a reminder that not everyone can be a competent and professional airline 
pilot. 


20.13. Maintenance of Manual Flying Skills 


It is agreed by about all that manual piloting skills degrade if they are not used. It can be 
somewhat distressing even to experienced pilots transitioning, for example, from a more 
advanced Boeing 777 aircraft to an earlier automation level aircraft like the Boeing 757, 
and observing that their basic instrument scanning skills have degraded. It is easy to 
become dependent on a primary flight display that basically displays all flight 
information during normal flight on one single display. In this case, the degradation is 
caused by a rusty instrument scan and is normally quickly recovered. The point is that 
there is a transition and that manual skills degrade when not used or when used, in this 
case, differently. 

Pilots are encouraged to use automation and most pilots like using automation—to 
varying degrees. The use of automation is supposed to reduce workload and allow 
pilots, especially in the busy terminal airspace, to monitor more effectively and to scan 
for other aircraft. Many airlines have procedures recommending use of automated 
systems and even requiring their use during certain conditions, as for a very low 
visibility approach and landing. Some airlines also train and discuss reducing levels of 
automation without losing some of the protective provisions that have been designed 
into the software. 

Pilots flying long-haul international routes face a special dilemma. They may be 
flying schedules that do not have many takeoffs and landings due to the long segment 
length. On the longer flight segments, the aircraft is scheduled with additional flight 
crew who serve as relief pilots and who are trained and rated in the aircraft. However, 
frequently there are simply not enough flight segments for all pilots to stay current with 
the FAA minimum of three takeoffs and landings within 90 days. Some airlines bring 
their long-haul flight crews back in the simulator every 90 days for a recurrent session. 
Some airlines have gone a step further and are keeping track of actual aircraft landing 
currency as opposed to landing currency met with a simulator session. 


20.14. Lacking a Good Foundation of Basic 
“Stick and Rudder” Skills 


What skills are we talking about? Do all pilots today enter the industry with a firm 
foundation of manual skills? Some of the multi-crew license (MPL) courses include a fair 
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portion of observer time as part of their curriculum. Is this time really the same as that of 
flying? Dr. Billings asks several prescient questions in his excellent book Aviation 
Automation: The Search for a Human-Centered Approach, which, although published in 
1997, seems eerily appropriate today: 


m Will pilots who have never had to acquire the finely tuned manual skills that older 
pilots take for granted be able to demonstrate such skills at an acceptable level if 
they must transition to another aircraft that lacks these advanced features? 


m Similarly, will they have learned the cognitive skills necessary for unassisted navi- 
gation if the flight management software fails? 


m Finally, and perhaps most important given the high reliability of today’s aircraft, 
will they acquire the judgmental skills and experience that alone can enable 
them to make wise decision in the face of uncertainty or serious mechanical or 
environment problems? (Billings, 1997) 


20.15. Two Examples of Superior Airmanship 


The Air Line Pilots Association, International (ALPA), which represents over 54,000 
pilots at 36 airlines, recently held its 55th Air Safety Forum and presented superior 
airmanship awards to two flight crews. It is interesting to note what types of skills 
contributed to the successful outcomes of these two events and to contemplate the role 
of training, standard procedures and experience. The first award was given to a Compass 
Airlines crew for an event occurring in May 2008. While in cruise flight at 32,000 feet, 
a warning message was displayed in the cockpit indicating a possible lavatory fire. 
Communication with the cabin was established; a flight attendant confirmed the 
existence of an active fire. Smoke began to enter the cockpit forcing the crew to don 
their oxygen masks. Unfortunately, due to an inadequately designed intercom system, 
communication with the cabin was then lost. The pilots reorganized the cockpit, 
declared an emergency and successfully made an emergency landing—in about eight 
minutes. Since they were unable to receive a response from the cabin crew, the pilots did 
not know if the fire had been put out or the general condition of the passengers. As it 
turned out, a flight attendant had been able to extinguish the fire and reseat the 
passengers away from the area. The crew did begin using some of their checklists. 
However, with concern that the airplane was literally burning up, they concentrated on 
flying the aircraft and getting the airplane on the ground and into the hands of the 
airport fire and rescue personnel. 
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The second Superior Airmanship award was presented to a United Airlines Airbus 
A-320 crew that suffered a partial electrical loss very shortly after takeoff from the New 
York area. There are six electronic displays in the A-320 cockpit. Half of them went blank 
in this incident, and several aircraft systems became inoperative. There are several 
particular operating notes for the NY area. First, especially in light of the events of 
September 11, 2001, there is a keen sensitivity to security events. Second, the New York 
area is very congested airspace. One of the consequences of this crew’s electrical failure is 
that they lost all radio communication. One of the concerns for all airline pilots today is 
that going “no-comm” will trigger the launch of military fighter jets that, if unable to 
establish communication and determine the aircraft to be a threat, may receive orders to 
shoot down the aircraft. To further complicate the situation, the United crew had 
degraded both instrumentation and warning systems. Similar to the Compass crew, they 
flew the aircraft, reorganized the cockpit and successfully returned to Newark for landing. 


20.16. Automation Does Not Reduce 
Training Requirements 


One of the great myths associated with increased automation in air transport operation is 
that automation reduces training. This is simply not true. Automation has created 
training requirements that add to the previous requirements. 

Increased automation is a welcome tool and is used by pilots to fly more safely 
and efficiently. However, the skills and knowledge associated with taking full advantage 
of increased automation must be added to the training curriculum. All of the former 
skills and knowledge of professional pilots are still required. 

It is simply not possible to monitor a system effectively without knowing how the 
system is planning to accomplish its task (Billings, 1991). Unfortunately, present training 
does not always ensure that pilots have this information. Recent accidents and incidents 
suggest that we do not have an adequate grasp of monitoring as it relates to human 
performance and limitations. 


20.17. Manual Skills Still Required—What 
Skills Constitute a “Proficient” Pilot? 


A high level of manual piloting skills is still required for today’s airline pilot. This means, 
almost by definition, that manual skills must be part of the training and evaluation 
program. There can be many reasons for skill deterioration, including such items as 
scheduling or even motivation, but in most cases of manual skills deterioration, the 
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problem is simply lack of practice or use. A great deal of the problem of maintaining 
manual skills can be minimized by company philosophy regarding the use of automation 
and emphasizing the importance of maintaining manual skills. Manual skill proficiency is 
something that should be evaluated as much as a pilot’s competence operating automated 
systems. 


20.18. Pilots Want and Need More System 
Knowledge 


At many airlines, pilots want and need more systems knowledge. They want to 
understand both how systems operate, and probably more importantly, the logic behind 
the design of the system. They do not want to be simply checklist readers and executors, 
always depending on a checklist or procedure to save them in an irregular or emergency 
operation. This would be true probably the majority of the time. But the dependence 
that it lends can be quite unhealthy. 

Pilots also complain about the reduction in “approved” training materials that are 
provided. Economics surely is a factor, but often the response from instructors or 
management is that, “if we put the material in the manual, then you are responsible in 
a checkride for that material.’ Certainly, this is the opinion of this author, but that 
viewpoint is neither acceptable nor satisfactory. Emergency and irregular operations, 
unfortunately, are not limited to “what’s in the manual.” The amount of systems material 
remains an issue, however, and one that warrants a mature discussion with pilots, trainers 
and oversight regulators. 


20.19. Intentional Non-Compliance 
with Procedures 


Unfortunately, even if training is improved, minimum standards raised, and more 
training materials provided, there remains the question, “Why do experienced and well- 
trained aircrews sometimes act against their experience and training and have 
accidents?” 

This question is not new. It has probably been around since Standard Operating 
Procedures (SOPs) were first developed and their use required. The Flight Safety 
Foundation Icarus Committee examined this question formally in 1992 at their first 
meeting. Their discussion of enabling factors, latent factors and, basically, what lies 
behind mistaken actions, resulted in the publication of 18 findings and ten 
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recommendations (Pinet and Enders, 1994). Orlady and Orlady discussed procedural 
non-compliance or the “willful deviator” in their book Human Factors for Multi-Crew 
Flight Operations. They list some common reasons as to why pilots choose not to comply 
with established standard operating procedures: 


m The pilot may think the established procedure is simply wrong. 


m The pilot may think the established procedures are okay for the “average” pilot, 
but that he/she is different. 


m The pilot may think his/her procedure is either just as good or better than the one 
established. 


m The pilot may think the procedure is not important or not necessary—or just not 
worth the bother—just this once, frequently, or always. 


m Lastly, in some instances, the pilot does not really object to the established 
procedure but consciously or subconsciously just wants to defy “them” —i.e., 
meaning management or authority in general. (Orlady and Orlady, 1999) 


It should be a priority for an airline both to minimize and to learn from non- 
compliance with established procedures and to incorporate what they learn into their 
training. It should also be a priority to communicate observations, findings and 
corrective actions developed with the pilots that may help them understand their own 
actions or those of another crewmember. As an example, this author’s briefing to her 
crewmembers on this topic is fairly simple: “if I’m doing something that is not SOP 
and I haven't told you about it, it probably is not intentional, so please speak up.” It 
remains one of this author’s pet peeves to read a company report describing an incident 
with a summary statement that “the crew failed to comply with SOPs.” Why? Was 
there an error of commission or omission based on an intentional action based on the 
list above? As Sidney Dekker and others have noted, there almost always is “more to 
the story” that a “failed to comply with SOPs” will not capture and, thus, miss 
a learning opportunity. 

SOPs, if they are sound, appropriate to the situation and developed with the input of 
those using them, are vital in today’s operations. The level of SOP non-compliance can 
also be a measure, frankly, of organizational health. That employees would adopt an 
attitude supporting “why bother having procedures if they are not to be followed” 
should surely trigger a re-examination of a number of basic tenets and influence of the 
“Pour Ps.” 
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20.20. Professionalism and Mentoring 


The topic of professionalism seems to quite be popular these days. It has been raised 
numerous times by the NTSB in recent accidents, not without some controversy. 
In October 2004, the Board listed the probable cause of the Pinnacle Airlines flight 
3701, which crashed on a repositioning flight, as “the pilots’ unprofessional 
behavior, deviation from standard operating procedures, and poor airmanship.” The 
probable cause statement for the Corporate Airlines crash in 2004 also contained 
the statement, “The pilots’ unprofessional behavior during flight....’ Similar 
inferences and attributions were made in both the 2006 Comair wrong runway 
takeoff accident in Lexington, Kentucky, and with the 2009 Colgan Buffalo crash. 
FAA Administrator Babbitt had an interesting statement in his opening remarks at 
ALPA’s recent 55th Air Safety Forum to an audience consisting primarily of airline 
pilots: 


We cannot regulate professionalism. No matter how many rules, regulations, advisories, 
mandatory training sessions, voluntary training sessions—pull them all together, and it still 
comes down to us—and by us, I mean every pilot...I’m not saying that pilots as a group are 
unprofessional. Quite the contrary. I am saying that this is a time for veterans to take the 
extra effort to mentor the pilots coming up through the ranks to ensure we maintain the 
highest levels of professionalism. This not only deals with safety, but with the need for you to 


help the new pilots learn how to react and adapt to change. 
(Babbitt, 2009) 


The ALPA Code of Ethics lists as a fundamental tenet the responsibility of an airline 
pilot to “remember that his/hers is a profession heavily dependent on training during 
regular operations, and, if in command, will afford his/her crew every reasonable 
opportunity, consistent with safety and efficiency, to learn and practice” (Air Line Pilots 
Association, 1956). However, it is important to recognize that professionalism and 
mentoring should be shared values in the airline and hopefully exemplified by 
management’s actions and behaviors. 


20.21. Another Type of Shakedown? 


A number of authors have written about the “shakedown” period that frequently 
accompanies the technological change that is definitional in the aviation industry. The 
shakedown refers to adverse consequences associated with the introduction and 
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adoption of new technologies. One pilot who soloed in the early 1940s and flew 
airliners for almost 40 years noted: 


One of the first things all of us need to remember is that change and technical innovation 
have always been with us. We have gone from lighted cans of kerosene to the first runway 
light; from the old loop and the Adcock four-course radio ranges to VORs and Cat II 
ILSs; from straight wings and the old suck, squeeze, bang and push piston engines to 
swept-wing airplanes and turbine power... There is nothing new about technological 


change. 
(Orlady, 1990a) 


One of the most pronounced shakedown periods came as pilots transitioned to the 
first jets in the aviation system. Not all were successful in their attempts; there were 
a fair number of initial rating ride failures on aircraft like the DC-8, B-720 and B-727. 
What is the relevance of the shakedown concept today? For all of the benefits and 
advantages of the advanced technology aircraft, is there the possibility that we have 
neglected to stress the importance of basic flying skills, situation awareness and pilot 
judgment? Are we seeing some sort of reverse shakedown or are we simply cutting too 
many corners with our training? Have these new technologies allowed pilots to 
develop an over-reliance and dependence instead of using the new technologies to 
assist them to better and more efficiently fly their aircraft? We must address these 
questions. 


20.22. Final Thoughts 


This chapter has attempted to briefly raise a number of critical and sensitive training 
issues. I recall a story often told by my late grandfather, Colonel Horace Orlady, who 
soloed in 1916 and flew in the US Signal Corp, the predecessor to the US Army. 
Their unit had a record of ordering an exceptionally high number of replacements 
parts (wings, landing gear, etc.), which, frankly, were tied to their dismal accident 
rate. An army efficiency expert was dispatched to address the issue. Observing so 
many aircraft in disrepair on his initial tour of the airfield, he queried the sergeant as 
to the cause. The sergeant succinctly responded, “bad landings, sir.’ The efficiency 
expert quickly addressed the situation with a large sign posted in the flight 
operations area the next day that stated, “There will be no more bad landings at this 


airfield.” 
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Unfortunately, we do not have a sign to post nor would anyone think that it would 
be an effective means to address what are complicated issues. We know a great deal more 
now about training, crew performance, system safety and the performance of highly 
reliable organizations. But we must continue efforts to understand how the different 
components of our aviation system work together, of the interface between the 
operators, machine and environment, and of the organizational influence on an 
operation. And we must continue efforts to protect and share voluntarily reported data 
so that others might learn. ICAO has mandated the implementation of safety 
management systems. SMS, while not expected to cure everything, should bring a new 
perspective and new tools as we try to manage the inherent risk in our operations. We 
still have lots to learn. 
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Introduction 


Predicting the future is a hazardous endeavor, but we can be sure that the scope of CRM 
for flight crews will be global, including threat and error management and operational 
assessment of crew practices through the Line Operations Safety Audit (LOSA) or 

a similar process. These are mandated for all the world’s airlines by the International 
Civil Aviation Organization (ICAO, 1998). The proliferation of CRM concepts in 
other industries will continue as more organizations and regulators concerned about 
safety and the prevention of harm recognize its value. While requiring CRM is an 
extremely important safety initiative, in practice it falls short of its intended goals. Major 
airlines in industrialized nations fulfill the requirements fully, recognizing the safety and 
operational benefits that accrue. However, some regional carriers and organizations in 
less developed nations are less punctilious in meeting the mandate. Effective CRM 
programs require resources that are in scarce supply in a time of economic distress. But 
we must also ask ourselves if we have been the CRM advocates we once were; not only 
through regulations, but by sharing lessons learned and success stories, and keeping the 
industry dialog active and relevant. 


21.1. Challenges 


Two events have had an enormous impact on aviation around the world: the terrorist 
attacks on the USA in 2001 and the global recession at the end of the decade. The 
Draconian increase in security, though necessary, has taken much of the pleasure out of 
flying. Long lines, forbidden items and questionable screening processes have made flying 
more onerous. One of the unfortunate effects has been on flight crew morale as all have to 
endure the same security screening process as passengers (RLH: I remember that when I 
got my first airline flight crew identification badge I was able to bypass security screening 
entirely). The recession has forced airlines into bankruptcy or mergers with other 
carriers. Pilots have suffered disrupted seniority which further harmed morale. Another 
potentially detrimental effect of recession may be on recruitment. Many of the best and 
brightest young aviators will probably be less likely to seek employment in an industry 
where the pay scale has deteriorated, pensions have been abrogated and layoffs loom. 
While it can be argued that the effective management of human error through CRM 
leads to the protection of assets and human life, the bid to stay afloat during harsh 
economic conditions may force even the most enlightened leadership to choose survival 
over CRM. This is more evident and pronounced in the face of global financial 
downturns when caretakers of governments and industry are faced with balancing 
dwindling economic resources, shrinking patronage as against meeting the minimum 
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requirements placed on maintaining CRM. This is why we feel that the transformation 
of CRM from merely a safety enhancement into a business necessity needs to be 
encouraged. 


21.2. CRM Beyond the Cockpit 


We have discussed the spread of CRM training into critical areas of airlines including 
maintenance and dispatch. CRM will continue to be introduced into other professions 
where teamwork is essential and team failures costly or deadly—medicine, firefighting, 
the rail and seafaring have already recognized the benefits of CRM and initiated training 
for those at the sharp end. We anticipate that the future will see refinements rather than 
generational shifts in the presentation of CRM to crewmembers. 

We have seen the evolution of CRM from the 1980s as a successful and responsible 
response to meet the ubiquitous consequences of human error. We hope that CRM will 
establish its legitimacy and solidify its place as a fundamental means of strengthening 
organizations against loss of assets, risk and the prevention of harm to human life. 

Training in coping with the myriad of threats to safety in the operational environ- 
ment will doubtlessly advance. While the introduction of automated, glass cockpit, 
aircraft led to worries about coordination with an electronic crewmember, organizations 
seem to have done this well and focused on managing the monitoring and sharing of 
operational information from within and outside the cockpit. The engineering 
community continues to shift its focus not only to the safety effects of technological 
advances by allowing crews to do what they do best—monitoring performance and 
recognizing threats to safety—but also to the demands of safeguarding ecology and 
making this a livable planet for many generations. While we are certain that successes in 
CRM, LOSA, FMAQ, ASAP and FOQA have established measurements towards the 
propensity to safety, the future may hold untapped value in ascertaining further effi- 
ciencies for monitoring crew actions that lead to environmental protection as a result of 
better communication and good decision-making at the corporate level. We now know 
that better crew functioning promotes technological advancements, improved fuel usage 
and efficient aircraft and powerplant design. We hope for a future where the application 
of CRM countermeasures not only combats human error, but could also provide the 
motivation for “earth-friendly” operations in safety critical industries. 


21.3. Operational Data 


Organizations will continue to need data about the practice of CRM in their own 
organization, not only for safety but also to keep training both current and relevant. 
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We have witnessed CRM applied worldwide as a basic initial or recurrent training 
program. This is a good starting point to consolidate concepts and deliberately 
enhance the awareness of crews about CRM (FAA, 2004). In most cases, these 
training programs have been modified to adapt manufacturers’ CRM training 
programs to reflect the local cultural milieu. Indeed, the localization of CRM could 
not have been realized without the inclusion of operational personnel at the sharp 
end: cabin crew, dispatchers, pilots and engineers—they are live sources of the data 
necessary to sustain CRM. Unfortunately, we have noted that CRM training that 
remains lodged in the classroom or simulator soon degenerates. It was not because 
they are wanting of fresh ideas but rather because these programs were bereft of 
operational data and did not have the capability to use CRM performance or risk 
measurements. The corpus of worldwide research unquestionably suggests that CRM 
training without the benefit of data and outcome measurement is doomed to fail. 
The Line Operations Safety Audit provides observational data on CRM practices. 
Surveys of crewmembers also provide useful feedback. Confidential incident 
reporting systems such as the Aviation Safety Reporting System, the Aviation Safety 
Action Program (ASAP) in the USA and the Confidential Human Factors Incident 
Reporting Programme (CHIRP) in the UK are extremely important sources of valid 
data on CRM and safety. 


21.4. Communities of Practice 


The personal sharing of issues and experiences in the CRM community had a big 
impact on advancing the state-of-the-art. In the 1980s and 1990s, annual CRM 
conferences were held which allowed airlines to discuss best practices in the delivery of 
training. This facilitated the innovation and refinement of CRM tools and techniques 
and served to help those operators who still lacked the organizational will to implement 
their own CRM programs. As CRM matured and budgets shrank, these meetings fell by 
the wayside. It is regrettable that this industry-wide sharing of methods and findings is 
no longer happening. 

However, in the mid 2000s, the FAA began to hold “Shared Vision” conferences 
which meet every two years. This conference reviews and reports on all of the FAA 
Voluntary Programs (see Farrow’s Chapter 14). It is not a dedicated meeting for CRM, 
but does include elements of CRM such as Advanced Qualification Program (AQP) and 
Line Operations Safety Audit (LOSA). Other voluntary programs (e.g. Aviation Safety 
Reporting System (ASRS), Aviation Safety Action Program (ASAP), Internal Evalua- 
tion Program (IEP)) generate operational data that can support CRM. Thus, the Shared 
Vision meetings provide an industry forum for CRM topics and the integration of such 
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programs into an operator’s Safety Management System (SMS) which will eventually 
become a requirement in the USA. In addition to the main benefit of bringing operators 
together to discuss their issues and experiences, the materials and directions are publicly 
available for worldwide benefit. 


21.5. Line Operational Simulation 


Line Oriented Flight Training (LOFT) which gives crews the opportunity to cope with 
non-standard or emergency situations in the simulator without fear of punishment 
continues to be one of the best venues for employing CRM and receiving feedback on 
crew performance. 

Over the last 15 years, we have seen the development of tools and methods that have 
improved and streamlined the development of LOS scenarios, training of instructor/ 
evaluators and refinement of event-set methodology to support reliable evaluation. 
Research and industry sharing of ideas and methods have contributed to this success 
story. In addition, Line Operational Simulation has provided an especially effective 
venue for introducing new aircraft, new technologies and new procedures in an 
operationally realistic setting. 

The use of line operational simulation is sure to provide continued benefit to 
crew training as the USA and other countries develop and implement Next 
Generation Air Transportation technologies (FAA, 2009). There will be a trans- 
formation of the National Airspace System (NAS) using new technologies to ensure 
that future safety, capacity and environmental needs are met. This transformation, 
which has already begun, will directly affect the tasks and procedures of air traffic 
controllers and pilots as well as the roles and responsibilities within and across ATC 
teams and flight crews. Major changes such as satellite-based rather than ground-based 
navigation and greatly enhanced information management systems will affect the 
entire aviation system. Just a few of the changes pilots anticipate are: new navigation 
procedures (e.g. RNAV, RNP, VNAV), new information sources (e.g. for weather, 
flight and airport information, traffic), and new information displays such as the 
Electronic Flight Bag which may contain a wide range of software options (e.g. 
charts, manuals, surface moving map, and various types of traffic information, ADS-B, 
CDTI). These changes are expected to overhaul where, when and how pilots access 
information, and they have the potential to enhance or undermine crew coordina- 
tion, communication, workload management and situational awareness. CRM and 
AQP can greatly facilitate the restructuring of flows and procedures during all phases 
of flight not only to maintain but to enhance the quality of crew performance with 
these new technologies. 
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21.6. New Challenges Ahead 


CRM training should be introduced during the earliest phases of flight training to 
ensure that the concept of crew coordination becomes ingrained in every aviator. The 
concept of the solo flight as a crowning achievement in training should be recognized as 
contrary to the concept of effective crew performance. There may be another unrec- 
ognized danger that presents a new challenge to CRM training. Just before the turn of 
the millennium because of the burgeoning availability of personal wealth, access to 
small, lightjet transport has become more accessible to an ever-growing entrepreneurial 
community. The affordable price tag of lightjets and the temptation of flying these jets 
without a costly personal pilot allow entrepreneurs such as doctors, lawyers, movie stars 
or Wall Street brokers to operate in airspace shared with commercial flights. Let alone 
the private pilot training that is required for the entrepreneur jet pilot, it is suggested 
that CRM training, which is most valuable in operating within shared airspace, may be 
wanting. Innovative ways may be needed to enable the aviation community to 
encourage CRM programs for the benefit of jet aircraft entrepreneurs. This is not to 
hint that CRM as we know it in commercial operations should be replicated for 
entrepreneur flying operations. There would be inconsistencies if CRM is mirrored to 
this type of operation. For instance, RVSM and TCAS operations that have commu- 
nications, decision-making and situational awareness impacts, though standardized 
across regulations, may be perceived differently by private and commercial pilots both 
operating high-performance jets in the same airspace. This is where the roles of 
regulators, the airlines, the research community and the public become more 
pronounced. 

Cultures, national, organizational and professional, will continue to be a major 
factor influencing the nature and delivery of CRM. This will remain a challenge 
for organizations in developing countries which may lack the resources to optimize 
training for their organizational and national cultures. The fact that CRM adapts 
well to diverse cultures needs to be emphasized in order for developing nations to 
shed off the notion that it is something “imported” from developed countries. 
CRM is best delivered as culture specific so long as it keeps its focus on mini- 
mizing the consequences of crew error. In yet another more interesting devel- 
opment, airlines have attempted to extend CRM into other non-operational 
departments. 

These initiatives are enviable and honestly heroic. 

Some have called these programs “Management Resource Management” or 
“Organizational Resource Management.” By whatever label, the future holds much 
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promise only if it is guided by learning from CRM implementation in its early 


stages: 


1. 


CRM understands the impossible condition of complete liberation from human 
error, therefore CRM in whatever variant must maintain its primordial focus in 
managing the effects of human error. 


. Threats and errors exist with certainty whenever humans operate complex 


systems. Understanding threats and errors with the appropriate and consistent 
application of CRM countermeasures alleviates and may prevent undesirable 
and unsafe situations. 


. CRM classroom training alone is lame and incomplete. The collection of threat 


and error data sustains effective application of CRM to generate planned action 
towards measuring performance and avoiding risk. 


. While CRM is concerned with operational safety and integrity, CRM designed 


for the corporation must be associated with commercial survival and integrity. 


- CRM success is attributed to the union of operational front-line and research 


inputs. Organizational CRM is not about mere team building or the estab- 
lishment of people efficiencies but rather a culmination of efforts drawn up to 
strengthen the resilience of an organization against asset loss and human harm. 


21.7. Final Word 


The authors in this volume exemplify the maturity of CRM and the diversity of its 
applications. While further venues will be found for CRM concepts, aviation will 
remain its cornerstone. As characterized by Cooke and Durso (2007), CRM is one of 
the success stories of modern psychology and cognitive engineering. While we may not 
be able to document statistically the number of lives saved by exemplary CRM practices 
and lives lost through CRM failures, the evidence for the value of CRM cannot be 
challenged. 
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